:W&Bá¾44îM s8hasm0rª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0à4–û‚" go8ly7zmq95i s8hasm0rŠ× s8hasm0rjuju11-uc-san-diegoSPAtop"š _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.03043084783154205 epochs‚10  hidden_dim‚128 ( learning_rate‚0.00027505904981709107  num_attention_heads‚8  num_branch_encoder_layers‚3  num_embedding_layers‚6  num_encoder_layers‚6Bjjtttt_vanilla_s8hasm0rb88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8Š  ÉãÔÕ¹‚š3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64ëÉ7 s8hasm0r&  _runtime‚0  _wandb.runtime‚0‘·`/jÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ŸÉãÔøÒ¯»*-of*=/data/spatop/options_files/sweep_tttt_jj/config_s8hasm0r.json*-l* /scratch/logs*-n*jjtttt_vanilla_s8hasm0r2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8r/opt/conda/bin/pythonº tghq7314jh57xmd50woxhe4riixy012cgNOÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ ”Ù°ª€àãÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-78f24734-e3a1-59a7-ce1b-bb4283f77c97Ê13.2º tghq7314jh57xmd50woxhe4riixy012cµª½Ç2  requirements.txt/3*`W s8hasm0rZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64Ý'äY s8hasm0rZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64!­ *  s8hasm0rb trainer/global_step2€ßÇ{\  s8hasm0rZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64@¾Äx/  s8hasm0rb*"trainer/global_step2[?J  s8hasm0rj9 ¡ÉãÔ°½•(*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ƒ(ø†“  s8hasm0rj  ¡ÉãÔ°é¿ î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 2.0 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 38.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ •G.ñ\ s8hasm0rjÊ ¡ÉãÔ ­¶¡·Trainable params: 40.9 M Non-trainable params: 86 Total params: 40.9 M Total estimated model params size (MB): 163.524 Modules in train mode: 4413 Modules in eval mode: 0 Total FLOPs: 0 æåF}q s8hasm0rj` ¡ÉãÔà£Î¡Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). ¯)·{ s8hasm0rjj ¡ÉãÔ€¥Ø¡Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). #Òëy s8hasm0rjh ¡ÉãÔøåß¡Xwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ÆF¶#€ s8hasm0rjo ¡ÉãÔðê¡_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). Õ.z s8hasm0rji ¡ÉãÔˆãú¡Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). QYH™t s8hasm0rjc ¡ÉãÔÈú‡¢Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). 5"–^q s8hasm0rj` ¡ÉãÔˆ•”¢Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). yÁt s8hasm0rjc ¡ÉãÔ¨œœ¢Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ÷·¤pn s8hasm0rj] ¡ÉãÔ¸Ò¤¢Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ½à'ám s8hasm0rj\ ¡ÉãÔØÍ°¢Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ¹¾;nb s8hasm0r*Ð  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"nR,Fë s8hasm0rjÙ ¡ÉãÔàƒ§È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. óLúN s8hasm0rj¼ ¡ÉãÔʪ§«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. z¥Ýoë s8hasm0rjÙ £ÉãÔÈؾÅÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. e=X5P s8hasm0rj¾ £ÉãÔðñÍÅ­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. á‘‚(:‡ °ÉãÔýóˆ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.1372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û‹„o‚(:‡ °ÉãÔ¬–î disk.rbd1.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚0) proc.memory.percent‚16.51477813720703 disk.sda.out‚ 0.5546875 disk.sda3.out‚ 0.5546875 disk.sda1.out‚0 proc.memory.rssMB‚ 2705.78125 disk.nvme0n1.out‚ 0.28515625 disk.sda.in‚0( proc.memory.availableMB‚ 366.83203125( disk./.usagePercent‚15.8498375929164 disk.nvme0n1.in‚0 disk.dm-1.out‚ 0.5546875 disk.sda2.out‚0% disk./.usageGB‚236.17403411865234 disk.sda3.in‚0 network.recv‚41177 disk.drbd1716.out‚0 disk.rbd1.out‚ 0.0859375 disk.dm-0.in‚0 disk.dm-0.out‚ 0.28515625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 network.sent‚33421 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0$ memory_percent‚97.76103496551514 cpu‚25.491233315494483 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0! disk.nvme0c0n1.out‚ 0.28515625 disk.drbd1716.in‚0Ÿ¢³q› ‚(:‘ ¿ÉãÔë»› gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.63' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.525999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0— =!‚(:³ ¿ÉãÔØ…  network.sent‚50580* disk./.usagePercent‚15.938804051177913 disk.sda.out‚ 1.17578125 disk.rbd0.in‚0 disk.sda1.out‚0! disk.dm-0.out‚482.76513671875% disk./.usageGB‚237.49969863891602 disk.sda3.out‚ 1.17578125& disk.nvme0c0n1.out‚482.76513671875 disk.dm-0.in‚ 0.4140625& proc.memory.availableMB‚ 363.921875 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 0.4140625 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.514897346496582 network.recv‚59713 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚ 0.4140625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.1484375# proc.memory.rssMB‚ 2705.80078125 disk.sda2.out‚0# memory_percent‚97.7787971496582 cpu‚25.37285531180284 disk.drbd2080.out‚0 disk.dm-1.out‚ 1.17578125$ disk.nvme0n1.out‚482.76513671875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0—;Äa3"b/ lr-AdamW/pg1"trainer/global_step2:JP<Ñsˆ3#b/ lr-AdamW/pg2"trainer/global_step2:JPÅß -$) _wandbruntime‚39  _runtime‚39ô¦æÂ%½ ' lr-AdamW/pg1‚1.1147968106730739e-05 ' lr-AdamW/pg2‚1.1147968106730739e-05 trainer/global_step‚49 % _timestamp‚1.7884049365923927e+09 _runtime‚ 39.580063503 _step‚0]½HkÄ&¿ ' lr-AdamW/pg1‚1.1147968106730739e-05 ' lr-AdamW/pg2‚1.1147968106730739e-05 trainer/global_step‚49 % _timestamp‚1.7884049365923927e+09 _runtime‚ 39.580063503 _step‚0!ýíÌl'bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP»„îñj(bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPŽ47%j)bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP¾Eâ¦`*b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPn÷Tj+bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP/-=o,bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPWÕ`-b\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPmIln.bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP^ÂëÓp/bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPÿÌ£m0bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPÐBl»k1bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP™› _q2bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP \à[j3bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP˜±ñl4bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP Á†Át5bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPæ•IW6bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP+E‰~j7bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP<|Am8bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP4õñf9bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP¶‹9Òe:ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPÔfÅxf;bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP~èô†eDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP7ù)l=bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPØ:/Åp>bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP>ôÚÈg?bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPÚ¹wi@be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP‘Â3[nAbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP þCkBbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP•Ø‡ÈiCbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPbœwiDbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP%Å*=fEbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPÔ§M>gFbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPÖRnGbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPÊ_fhHbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP6Z>)mIbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP÷!nJbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP³hëìiKbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPä'ºmLbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP½Ã.ÏdMb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP¿F2ýjNbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP¶airObn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP÷¾þ_Pb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPëÀsxpQbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPE¢'égRbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP˜ãQVhSbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPlC%¹qTbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPcñTsUbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP 4“wnVbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPoî<—fWbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP^alXbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP&—HkcYb_ idbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP7¥«ìoebk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP—ÖÌ-bfb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPŒž`%pgbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP?ÁkÛthbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP­¿d3fibb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPM½tóijbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPê(sèkkbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP0KIclb_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPRãgzl„bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP †öf…ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP¢uÍE^†bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPGàªAr‡bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP3¾Žøpˆbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPé’×…h‰bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPÛ4ì¶`Šb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPé.Öi‹bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPÄ ýšgŒbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP.x0sbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPiû9…bŽb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPìZmbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP ‹yWibd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPâç“Ah‘bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP‹]y’bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP’B¿c“b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPÊKŽÎe”b` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPó–‚îf•ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPŒÂJr–bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPãrÚþl—bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPklmm˜bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP‘ˆtOn™bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP«ð¤³tšbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPõÞ‚j›be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPZ’ˆïhœbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP-ÅÒ:tbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP¾7Þqžbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPö| rŸbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPùEõ$p bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP3¼mw¡br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP j‰•.¢) _wandbruntime‚39  _runtime‚39`Á2œ£üN \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2875955288576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚921198 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2760889916416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚939738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚939738 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3103168254464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3146520891904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚854 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2834546425344 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 41390860288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954818048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845965 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 _runtime‚ 39.580938944 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚708396 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2801473167074 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚708398 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954818048 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚708396 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚231480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚843 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2875281690112 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 40583250658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚843 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚921198 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 40735264768 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2875281690112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 43517028352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚843 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 181444096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691469 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 2760906955776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3146709222912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008ºßï݆'¤€O SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2801473167074 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚708398 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954818048 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚708396 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚231480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚843 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2875281690112 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 40583250658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚843 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚921198 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 40735264768 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2875281690112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 43517028352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚843 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 181444096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691469 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 2760906955776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3146709222912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2875955288576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚921198 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2760889916416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚939738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚939738 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3103168254464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3146520891904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚854 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2834546425344 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 41390860288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954818048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845965 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 _runtime‚ 39.580938944 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚708396 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952¡Ÿf¥ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPÍìÒm¦bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPº©d¦r§bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP÷1øßg¨bb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP]½ ¸p©bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPn8nªbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPX–j«be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPÌ»WŸj¬be BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JP…s¼l­bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPt.¥j®be BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPß2h¯bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP};I¾e°b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP¨<7—]±bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPH•[Œk²bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPj‘¯Öu³bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP¾ôf4h´bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPýmqµbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP±HÛg¶bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP­Ìnëk·bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPý$¾œg¸bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPXϼøf¹ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP´;×2iºbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP\Ö„m»bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPšsúr¼bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP@kCb½b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPý¨d"q¾bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPß'mpl¿bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPØ µ]ÀbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPÍÓºVÁbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPW–œsiÂbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPšUX=dÃb_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP e5>oÇbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP™v–^ÈbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPÕwh¸bÉb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP[w>5iÊbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPfÁˆ:mËbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP»ªi8kÌbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPz·mÍbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP–l”hÎbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPI97rÏbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP ÚM\ÐbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPx)ØgÑbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPIò “pÒbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPöÑß/fÓba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP ›,lÔbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP*Ú×[oÕbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPqµõoÖbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPvÈçôm×bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP¿kødlØbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPD=?^jÙbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPEgˆkÚbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPÖBÌjÛbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP¤¶ÓuÜbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPš¢9ylÝbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPüo¸mÞbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP÷„·mhßbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JP:ñ´ždàb_ ,òFeðb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP<ƒò_cñb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPÓûÎqbòb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPyÄƬwóbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPØëZJrôbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPTþ¯fõba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPÐq¥eöb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP££O_d÷b_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPãghúbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPÅsX’oûbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPXâ¼mübh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP78Ë~rýbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP[³Ý2cþb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPï ‰'eÿb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPʨAe€b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPŽ É–_bZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JPr'¥úm‚bh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPŽ$M@mƒbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP‡d¡q„bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPðûyns…bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP¼ÍÉw†br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP ¶Ösm‡bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPHü{áaˆb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPÃIUse‰b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPÃÄ©dŠb_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP&j;ùeŽb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPä&“pbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPcJlhbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JP‡%{ø_‘bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPlïzÈj’be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP¡C k“bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP‰Y--i”bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPæÜaf•ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP£GRPn–bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPô‘]k—bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPD•x™`˜b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPñ[êd™b_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚939652 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2933127768576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 41554692608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2890917467136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2890899464192 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚658355 $ _timestamp‚1.788404937286694e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10192158720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 41554692608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2890899464192 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2932454156800 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚722412 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 42054116038 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚422270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚235630 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.currØ·×ßVent‚457 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 181444096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 42210301440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2857865202374 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 trainer/global_step‚49 _step‚2 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚722414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1662064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3164842932224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚722412 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚657898 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2933127768576 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2890917467136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 42210301440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 _runtime‚ 40.274564071 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚958218üAÇÀ&¢ŠM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 trainer/global_step‚49 _step‚2 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚722414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1662064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3164842932224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚722412 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚657898 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2933127768576 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2890917467136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 42210301440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 _runtime‚ 40.274564071 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚958218 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚422268 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚236085 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3164818992128 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 44373842432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚939652 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2933127768576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 41554692608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2890917467136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2890899464192 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚658355 $ _timestamp‚1.788404937286694e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10192158720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 41554692608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2890899464192 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2932454156800 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚722412 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 42054116038 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚422270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚235630 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 181444096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 42210301440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2857865202374 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904߈TLA£b< loss/FRt2/assignment_loss"trainer/global_step2:JPyùA¤b< loss/FRt4/assignment_loss"trainer/global_step2:JP¡|̪C¥b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP¹ßm loss/SRqqt2/assignment_loss"trainer/global_step2:JPƒ‚m§B³b= loss/SRqqt3/detection_loss"trainer/global_step2:JPHór‰B´b= loss/SRqqt4/detection_loss"trainer/global_step2:JPl±EAµb< loss/FRt1/assignment_loss"trainer/global_step2:JPñjwˆA¶b< loss/FBt4/assignment_loss"trainer/global_step2:JPÅS>D@·b; loss/FRt2/detection_loss"trainer/global_step2:JPmðƒ«C¸b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP'ÈC¹b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPÔ¬õ7ºb2 loss/total_loss"trainer/global_step2:JP£Õ@»b; loss/FRt1/detection_loss"trainer/global_step2:JP‘_%Ý@¼b; loss/FBt4/detection_loss"trainer/global_step2:JPå¸{–.½) _wandbruntime‚40  _runtime‚40ÇBI¾à 1loss/SRqqt1/assignment_loss‚5.411560811494526 0loss/FBt3/assignment_loss‚2.7305754282895256 /loss/FRt3/detection_loss‚3.9337041773075283 0loss/SRqqt1/detection_loss‚4.316772939343202 1loss/SRqqt2/detection_loss‚2.7830805169828867 0loss/FRt3/assignment_loss‚7.5225636238275575 0loss/FBt1/assignment_loss‚2.7745189666748047 0loss/FBt2/assignment_loss‚2.7626332496774606 _runtime‚ 40.275981675 .loss/FBt3/detection_loss‚4.890598150737145 .loss/FRt4/detection_loss‚4.693504338328903 .loss/FBt1/detection_loss‚5.994295294529626 epoch‚0 .loss/FBt2/detection_loss‚6.109671689313034 1loss/SRqqt2/assignment_loss‚5.352425777305991 /loss/SRqqt3/detection_loss‚2.17458682049598 0loss/SRqqt4/detection_loss‚5.095770666704459 0loss/FRt1/assignment_loss‚7.3010636329650875 /loss/FBt4/assignment_loss‚2.701360826139097 /loss/FRt2/detection_loss‚2.9383603879115845 1loss/SRqqt3/assignment_loss‚5.532995687212263 1loss/SRqqt4/assignment_loss‚5.498057295294369 &loss/total_loss‚117.44057074623724 .loss/FRt1/detection_loss‚5.868603207667669 .loss/FBt4/detection_loss‚6.511116958878659 trainer/global_step‚49 % _timestamp‚1.7884049372885325e+09 _step‚3 /loss/FRt2/assignment_loss‚7.149043745464748 /loss/FRt4/assignment_loss‚7.3937065536911430«É£M¿Ç .loss/FRt4/detection_loss‚4.693504338328903 .loss/FBt1/detection_loss‚5.994295294529626 epoch‚0 .loss/FBt2/detection_loss‚6.109671689313034 1loss/SRqqt2/assignment_loss‚5.352425777305991 /loss/SRqqt3/detection_loss‚2.17458682049598 0loss/SRqqt4/detection_loss‚5.095770666704459 0loss/FRt1/assignment_loss‚7.3010636329650875 /loss/FBt4/assignment_loss‚2.701360826139097 /loss/FRt2/detection_loss‚2.9383603879115845 1loss/SRqqt3/assignment_loss‚5.532995687212263 1loss/SRqqt4/assignment_loss‚5.498057295294369 &loss/total_loss‚117.44057074623724 .loss/FRt1/detection_loss‚5.868603207667669 .loss/FBt4/detection_loss‚6.511116958878659 trainer/global_step‚49 % _timestamp‚1.7884049372885325e+09 _step‚3 /loss/FRt2/assignment_loss‚7.149043745464748 /loss/FRt4/assignment_loss‚7.393706553691143 1loss/SRqqt1/assignment_loss‚5.411560811494526 0loss/FBt3/assignment_loss‚2.7305754282895256 /loss/FRt3/detection_loss‚3.9337041773075283 0loss/SRqqt1/detection_loss‚4.316772939343202 1loss/SRqqt2/detection_loss‚2.7830805169828867 0loss/FRt3/assignment_loss‚7.5225636238275575 0loss/FBt1/assignment_loss‚2.7745189666748047 0loss/FBt2/assignment_loss‚2.7626332496774606 _runtime‚ 40.275981675 .loss/FBt3/detection_loss‚4.890598150737145†^XÐ’À‚(:‡ ÎÉãÔìÅ¢ gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«’î£EÁ‚(:º ÎÉãÔïdž  network.sent‚69902 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.16796875 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.dm-0.in‚ 4.16796875 disk.dm-1.in‚0 disk.sda3.out‚ 1.80859375 disk.drbd2266.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1361.46337890625 disk.sda2.out‚0 disk.sda3.in‚0% disk./.usageGB‚237.50040435791016" disk.dm-0.out‚1361.46337890625 disk.sda2.in‚0$ memory_percent‚97.77936935424805# proc.memory.rssMB‚ 2705.83203125 disk.dm-1.out‚ 1.80859375 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 0.2421875 cpu‚25.30308101515623 network.recv‚80165 disk.nvme0n1.in‚ 4.16796875% disk.nvme0n1.out‚1361.46337890625 disk.sda.out‚ 1.80859375 disk.drbd2080.out‚0* proc.memory.percent‚16.515088081359863* disk./.usagePercent‚15.938851412571747 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 363.828125 disk.rbd1.in‚0ؤ¦Æ‚(:’ ÝÉãԯ̫ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.999' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.599799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xŠå¤<Â(:± ÝÉãÔØ­Œ * proc.memory.percent‚16.515135765075684 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda.out‚2.328125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.16796875 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 363.31640625 disk.sda2.in‚0 disk.rbd1.out‚ 0.3046875 disk.dm-0.in‚ 4.16796875 disk.drbd2265.in‚0 network.sent‚133556) disk./.usagePercent‚15.93884373234572$ disk./.usageGB‚237.5002899169922 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.out‚2.328125 disk.drbd2266.out‚0 proc.cpu.threads‚17 network.recv‚111666 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚2.328125 cpu‚25.257789980152264 disk.dm-0.out‚1362.251953125$ memory_percent‚97.78249263763428# proc.memory.rssMB‚ 2705.83984375 disk.nvme0c0n1.in‚ 4.16796875% disk.nvme0c0n1.out‚1362.251953125# disk.nvme0n1.out‚1362.251953125‰5z’Ä‚(:‡ ìÉãÔ÷¥Œ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÃJî:Å‚(:¯ ìÉãÔã¡Ü $ memory_percent‚97.77967929840088 network.recv‚116509 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 4.16796875% disk.nvme0c0n1.out‚1362.607421875 disk.drbd1716.out‚0# disk.nvme0n1.out‚1362.607421875* proc.memory.percent‚16.515231132507324 cpu‚25.22485151803329 disk.sda3.out‚3.03125 disk.nvme0n1.in‚ 4.16796875 disk.dm-0.in‚ 4.16796875 disk.dm-1.out‚3.03125# proc.memory.rssMB‚ 2705.85546875 proc.cpu.threads‚17 disk.dm-0.out‚1362.607421875 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.sda.in‚0* disk./.usagePercent‚15.938810451366269 disk.rbd1.in‚0 disk.rbd1.out‚ 0.3046875 disk.drbd2265.out‚0 disk.sda.out‚3.03125( proc.memory.availableMB‚ 363.77734375 disk.sda2.out‚0 disk.drbd2265.in‚0 network.sent‚140173 disk.drbd2080.out‚0 disk.sda1.out‚0% disk./.usageGB‚237.49979400634766 disk.rbd0.out‚0ç3Uw.Æ) _wandbruntime‚76  _runtime‚76Iƒ~Ãǽ ' lr-AdamW/pg2‚2.2523445766660063e-05 trainer/global_step‚99 % _timestamp‚1.7884049732197466e+09 _runtime‚ 76.207202414 _step‚4 ' lr-AdamW/pg1‚2.2523445766660063e-05–þoÇÈÁ ' lr-AdamW/pg1‚2.2523445766660063e-05 ' lr-AdamW/pg2‚2.2523445766660063e-05 trainer/global_step‚99 % _timestamp‚1.7884049732197466e+09 _runtime‚ 76.207202414 _step‚4&·³·.É) _wandbruntime‚76  _runtime‚76ýx QN#ʇO SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1409198 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10192158720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3256446 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5652198367744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3274988 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5587515034482 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚843 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1409198 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6271899681792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 86194544128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚822502 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚ 76.208139952 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5734593036800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5652216370688 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5733919438336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5506096444416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151808000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5733919438336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 82376666112 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1409196 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1409196 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 81721070592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6185729077760 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5588186504050 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 81418590066 % _timestamp‚1.7884049732204854e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1865790 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚413 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5652216370688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5506113483776 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 82073020274 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1847250 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6272075429888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚459659 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6185705137664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚843 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚459222 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 trainer/global_step‚99 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5652198367744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183540224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚822500 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5734593036800 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 81721070592 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1865790 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1847250 PDDeviceStatsMonitor.on_k‰Ëº?train_batch_start/inactive_split.all.allocated‚1282161 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚863 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 82376666112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413N÷•=‘'Ë‹O TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10192158720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3256446 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5652198367744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3274988 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5587515034482 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚843 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1409198 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6271899681792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 86194544128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚822502 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚ 76.208139952 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5734593036800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5652216370688 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5733919438336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5506096444416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151808000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5733919438336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 82376666112 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1409196 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1409196 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 81721070592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6185729077760 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5588186504050 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 81418590066 % _timestamp‚1.7884049732204854e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1865790 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚413 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5652216370688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5506113483776 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 82073020274 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1847250 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6272075429888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚459659 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6185705137664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚843 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚459222 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 trainer/global_step‚99 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5652198367744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183540224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚822500 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5734593036800 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 81721070592 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1865790 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1847250 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1282161 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚863 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 82376666112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1409198 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360™4î÷.Ì) _wandbruntime‚76  _runtime‚76“±É—&Í‘M ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5561000574976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10192158720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 76.902409254 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 82235025030 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚830670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 82540498432 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1865704 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚464205 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1294875 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3307484 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5708569409536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 87047887872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1294420 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1423214 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚463752 _step‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5708569409536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5561017614336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1423212 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5643907069782 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 87216459776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183540224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10192158720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6334403763200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3288916 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 866123776 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3288916 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5791091905024 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 trainer/global_step‚99 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 83196107264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6247379815424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1423214 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1884270 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5791765516800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1884270 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5791765516800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5708551406592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 332015104 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6334596275200 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6247355875328 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5643235600006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6334403763200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3288916 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 866123776 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3288916 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5791091905024 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 trainer/global_step‚99 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 83196107264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6247379815424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1423214 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1884270 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5791765516800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1884270 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5791765516800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5708551406592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 332015104 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6334596275200 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6247355875328 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5643235600006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 82235025030 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚830670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 82540498432 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1865704 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚464205 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1294875 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3307484 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5708569409536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 87047887872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1294420 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1423214 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚463752 _step‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5708569409536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5561017614336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1423212 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5643907069782 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 87216459776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183540224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10192158720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 122908359682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2773302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2109998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2773302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 123362471936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚149 _runtime‚ 113.008840641 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8469868313088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8469850310144 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2791842 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 123362471936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8593230785024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8469850310144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 122706876416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _step‚9 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8593230785024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8592557186560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2109996 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚687885 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚687447 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 129289340928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928(8xœœ'Û–O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _step‚9 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8593230785024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8592557186560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2109996 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚687885 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚687447 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 129289340928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 122706876416 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8374228371458 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2791842 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9268242020864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8373556901890 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1918347 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9268265960960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 8251302972416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9397364873728 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4883298 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 122908359682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2773302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2109998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2773302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 123362471936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚149 _runtime‚ 113.008840641 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8469868313088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8469850310144 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2791842 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 123362471936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8593230785024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8469850310144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 122706876416 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8650403265024 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2124012 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2791756 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 123070364438 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1239068 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9329916698624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8526221351936 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1930972 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196706304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8306207102976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 129974044160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4915768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2791756 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 123526304256 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1931412 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 124181913088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9329892758528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9459866802688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 123526304256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8526203348992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚10 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small'ÚûÐ_pool.allocated‚ 123724794854 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8650403265024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124181913088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 113.699486195 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2124012 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2810322 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚691904 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9460063508992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚149 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1239070¼O‡u¨&Þ¢M D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 124181913088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9329892758528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9459866802688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 123526304256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8526203348992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚10 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 123724794854 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8650403265024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124181913088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 113.699486195 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2124012 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2810322 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚691904 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9460063508992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚149 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1239070 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8429948937190 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8650403265024 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2124012 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2791756 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 123070364438 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1239068 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9329916698624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8526221351936 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1930972 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196706304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8306207102976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 129974044160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4915768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2791756 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 123526304256 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1931412 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 Öž-g0ß+ _wandbruntime‚113  _runtime‚113‰<€¯Iàà /loss/FRt1/assignment_loss‚7.274203245980399 0loss/FBt4/assignment_loss‚0.9667740834170374 $loss/total_loss‚130.339015514352 /loss/FRt2/assignment_loss‚7.013614165782928 0loss/FBt2/assignment_loss‚0.9996728849411011 1loss/SRqqt2/detection_loss‚2.8396626062415264 /loss/SRqqt4/detection_loss‚7.73932091078975 /loss/FRt3/assignment_loss‚7.398130705190259 .loss/FRt1/detection_loss‚4.853572110618864 .loss/FBt2/detection_loss‚7.107547018527985 .loss/FBt4/detection_loss‚2.052487647334507 1loss/SRqqt2/assignment_loss‚5.141003529230754 1loss/SRqqt3/assignment_loss‚5.424791515574736 0loss/SRqqt1/detection_loss‚3.653265270259645 /loss/FBt3/detection_loss‚14.881971690803766 trainer/global_step‚149 /loss/FRt4/assignment_loss‚7.312439764247221 0loss/FBt3/assignment_loss‚1.0952290445566177 .loss/FRt4/detection_loss‚4.932256534695625 /loss/FRt3/detection_loss‚3.9988572652949843 1loss/SRqqt3/detection_loss‚1.2299236051628695 /loss/FBt1/detection_loss‚19.325708899233078 % _timestamp‚1.7884050107131479e+09 _runtime‚ 113.70051747 1loss/SRqqt4/assignment_loss‚5.339678590947932 .loss/FRt2/detection_loss‚3.167849711794406 epoch‚0 _step‚11 1loss/SRqqt1/assignment_loss‚5.313582727644179 /loss/FBt1/assignment_loss‚1.277471986081865Û’•MáÇ .loss/FBt2/detection_loss‚7.107547018527985 .loss/FBt4/detection_loss‚2.052487647334507 1loss/SRqqt2/assignment_loss‚5.141003529230754 1loss/SRqqt3/assignment_loss‚5.424791515574736 0loss/SRqqt1/detection_loss‚3.653265270259645 /loss/FBt3/detection_loss‚14.881971690803766 trainer/global_step‚149 /loss/FRt4/assignment_loss‚7.312439764247221 0loss/FBt3/assignment_loss‚1.0952290445566177 .loss/FRt4/detection_loss‚4.932256534695625 /loss/FRt3/detection_loss‚3.9988572652949843 1loss/SRqqt3/detection_loss‚1.2299236051628695 /loss/FBt1/detection_loss‚19.325708899233078 % _timestamp‚1.7884050107131479e+09 _runtime‚ 113.70051747 1loss/SRqqt4/assignment_loss‚5.339678590947932 .loss/FRt2/detection_loss‚3.167849711794406 epoch‚0 _step‚11 1loss/SRqqt1/assignment_loss‚5.313582727644179 /loss/FBt1/assignment_loss‚1.277471986081865 /loss/FRt1/assignment_loss‚7.274203245980399 0loss/FBt4/assignment_loss‚0.9667740834170374 $loss/total_loss‚130.339015514352 /loss/FRt2/assignment_loss‚7.013614165782928 0loss/FBt2/assignment_loss‚0.9996728849411011 1loss/SRqqt2/detection_loss‚2.8396626062415264 /loss/SRqqt4/detection_loss‚7.73932091078975 /loss/FRt3/assignment_loss‚7.398130705190259 .loss/FRt1/detection_loss‚4.853572110618864 Þâ‚(:’ ™ÊãÔ“ð… gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.999' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.599800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/à}OKã‚(:À ™ÊãÔ“áô  proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2705.87109375 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd2265.out‚0$ memory_percent‚97.78223037719727& proc.memory.availableMB‚ 363.359375 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.rbd1.in‚0 disk.sda.out‚ 4.90234375 disk.drbd2080.out‚0 network.recv‚132854 disk.sda3.out‚ 4.90234375 disk.drbd2080.in‚0' disk.nvme0c0n1.out‚1364.75634765625 disk.rbd1.out‚ 0.5556640625 disk.dm-0.in‚ 4.16796875 network.sent‚203950 disk.rbd0.out‚0 cpu‚25.170539259715195 disk.sda1.in‚0* disk./.usagePercent‚15.938711376450518% disk./.usageGB‚237.49831771850586" disk.dm-0.out‚1364.75634765625 disk.drbd2265.in‚0* proc.memory.percent‚16.515326499938965 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.16796875% disk.nvme0n1.out‚1364.75634765625 disk.dm-1.out‚ 4.90234375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0(Ç< ’ä‚(:‡ ¨ÊãÔé® gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈçGýFå‚(:» ¨ÊãÔ韛 $ disk./.usageGB‚237.4983367919922 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0$ memory_percent‚97.78158664703369 network.recv‚138019 disk.sda.in‚0 disk.sda.out‚ 5.38671875 disk.sda3.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚1364.97119140625 disk.rbd0.out‚0 proc.memory.rssMB‚2705.875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.16796875 disk.dm-0.in‚ 4.16796875" disk.dm-0.out‚1364.97119140625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.drbd1716.in‚0% disk.nvme0n1.out‚1364.97119140625 disk.dm-1.out‚ 5.38671875( proc.memory.availableMB‚ 363.46484375* proc.memory.percent‚16.515350341796875* disk./.usagePercent‚15.938712656488189 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 5.38671875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 0.5556640625 network.sent‚253216 disk.drbd2266.out‚0 disk.drbd2080.in‚0 cpu‚25.15757063614429ï-[0æ+ _wandbruntime‚149  _runtime‚149Õë+ëÆçÀ _runtime‚ 149.768874185 _step‚12 ' lr-AdamW/pg1‚4.5274401086518714e-05 ' lr-AdamW/pg2‚4.5274401086518714e-05 trainer/global_step‚199 % _timestamp‚1.7884050467813694e+09®”‹ ÊèÄ _runtime‚ 149.768874185 _step‚12 ' lr-AdamW/pg1‚4.5274401086518714e-05 ' lr-AdamW/pg2‚4.5274401086518714e-05 trainer/global_step‚199 % _timestamp‚1.7884050467813694e+09 ÆŽc0é+ _wandbruntime‚149  _runtime‚149‰_ïª'ê¤O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1639302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12350778904064 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 171953641984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2810798 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3717894 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 172118032896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2810798 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 163692682240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 163089268882 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11159598769298 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 163692682240 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2555783 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _runtime‚ 149.769761849 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2810796 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2555331 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚916031 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12522920877056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 163743699090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚916481 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3717894 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11451868533248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11287502252544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10996509500416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11287520255488 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11451194934784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12350802844160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10996526539776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11287502252544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2810796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 164348277760 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11451868533248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11287520255488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12522732546048 % _timestamp‚1.7884050467821176e+09 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11451194934784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6528692 _step‚13 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3699354 DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11287502252544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2810796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 164348277760 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11451868533248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11287520255488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12522732546048 % _timestamp‚1.7884050467821176e+09 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11451194934784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630f;<1ö ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6528692 _step‚13 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3699354 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1639302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12350778904064 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 171953641984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2810798 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3717894 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 172118032896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2810798 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 163692682240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 163089268882 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11159598769298 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 163692682240 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2555783 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _runtime‚ 149.769761849 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2810796 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2555331 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚916031 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12522920877056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 163743699090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚916481 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3717894 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11451868533248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11287502252544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10996509500416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 &jK’ì‚(:‡ ·ÊãÔŸ¾‡ gpu.0.gpu‚18.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.728' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±f׎?í‚(:´ ·ÊãÔÓ¥Š  disk.rbd1.out‚ 0.650390625% disk.nvme0n1.out‚1365.62548828125 disk.dm-0.in‚ 4.16796875 network.sent‚259837 network.recv‚142644 disk.drbd2080.in‚0' disk.nvme0c0n1.out‚1365.62548828125 disk.drbd1716.in‚0* disk./.usagePercent‚15.938720848729284 disk.drbd2080.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.515398025512695 cpu‚25.148693093964347$ disk./.usageGB‚237.4984588623047 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2705.8828125 proc.cpu.threads‚17" disk.dm-0.out‚1365.62548828125 disk.sda3.in‚0 disk.sda3.out‚6.03125 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0$ memory_percent‚97.78203964233398 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.nvme0n1.in‚ 4.16796875& proc.memory.availableMB‚ 363.390625 disk.drbd2265.out‚0 disk.sda.out‚6.03125 disk.dm-1.in‚0 disk.dm-1.out‚6.03125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0—ä}0î+ _wandbruntime‚150  _runtime‚150©‰( ´&ï®M B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12585235206144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11215990804598 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 172978330624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3717808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚ 150.466533412 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2824812 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2568073 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2824814 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12412429641728 _step‚14 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 164560134262 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3736374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚920605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2824814 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2568533 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 165167718912 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 172805564416 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11343873294336 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3717808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11051430670336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6561188 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 163905703846 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6542620 % _timestamp‚1.7884050474784544e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11051413630976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 164512110080 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11509041013248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 164512110080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12585431912448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6542620 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1647468 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚921063 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12412453581824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172766208 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11215319334822 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1647470 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2824812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11343855291392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11508367401472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11343855291392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952ˆŸê\¸&ð²M C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11508367401472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11343855291392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12585235206144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11215990804598 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 172978330624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3717808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚ 150.466533412 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2824812 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2568073 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2824814 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12412429641728 _step‚14 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 164560134262 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3736374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚920605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2824814 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2568533 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 165167718912 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 172805564416 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11343873294336 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3717808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11051430670336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6561188 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 163905703846 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6542620 % _timestamp‚1.7884050474784544e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11051413630976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 164512110080 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11509041013248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 164512110080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12585431912448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6542620 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1647468 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚921063 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12412453581824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172766208 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11215319334822 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1647470 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2824812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11343855291392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚860 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8155544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 215005637120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4643946 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203924608290 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 204678488064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚879 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚17 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4625406 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1146382 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 204678488064 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3511598 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3511596 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14309832683008 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4625406 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047700 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15648159130624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15433339727360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3511598 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14309832683008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884050833165886e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8137002 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13741716028416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14105154194944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542ïDc[¯'ý©O YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 204678488064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚879 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚17 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4625406 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1146382 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 204678488064 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3511598 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3511596 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14309832683008 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4625406 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047700 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15648159130624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15433339727360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3511598 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14309832683008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884050833165886e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8137002 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13741716028416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14105154194944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14105154194944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13946312106274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13741733067776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14105172197888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2047702 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14105172197888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15648345364480 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3194523 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15433315787264 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 214843343360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 205334083584 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3194082 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 204579038498 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1146821 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚860 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8155544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 215005637120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4643946 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203924608290è+0þ+ _wandbruntime‚186  _runtime‚186ñ³+¶&ÿ°M XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 206153524736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 205395473670 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14161507233792 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3206851 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 215693541376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4643860 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4662426 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1150983 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 186.990154969 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3525614 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3525612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151422 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 205497915904 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204741043254 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14367678761472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2055870 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ræm<€´M XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 206153524736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 205395473670 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment;`T~ .large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14161507233792 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3206851 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 215693541376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4643860 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4662426 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1150983 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 186.990154969 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3525614 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3525612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151422 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 205497915904 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204741043254 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14367678761472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2055870 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚09M„‹0+ _wandbruntime‚186  _runtime‚186üðxU‚Ï 0loss/FBt3/assignment_loss‚0.5904110497639825 1loss/SRqqt2/detection_loss‚1.9510471010437378 /loss/FBt3/detection_loss‚5.4733860721190775 &loss/total_loss‚114.48791023151881 epoch‚0 /loss/FRt2/detection_loss‚2.5865663781762125 _runtime‚ 186.991532427 /loss/FRt2/assignment_loss‚7.027908885478974 1loss/SRqqt3/detection_loss‚1.5579218879145404 trainer/global_step‚249 2loss/SRqqt2/assignment_loss‚3.1437744250664346 .loss/FRt4/detection_loss‚3.946899793868841 .loss/FBt1/detection_loss‚34.13238373398781 .loss/FBt2/detection_loss‚4.450126463497007 /loss/FBt4/detection_loss‚1.2710481541877836 2loss/SRqqt1/assignment_loss‚3.4306601773608816 0loss/FBt1/assignment_loss‚0.9014526844024658 1loss/FBt2/assignment_loss‚0.34578847647267014 0loss/FBt4/assignment_loss‚0.6528771820069268 0loss/SRqqt4/detection_loss‚3.209837111283322 /loss/FRt1/assignment_loss‚7.255180707344642 0loss/FRt3/assignment_loss‚7.4015690722364065 2loss/SRqqt4/assignment_loss‚3.6500508104051863 1loss/SRqqt1/detection_loss‚3.6235787238587034 _step‚19 % _timestamp‚1.7884050840041738e+09 /loss/FRt4/assignment_loss‚7.338856497476267 .loss/FRt1/detection_loss‚3.232156440615654 .loss/FRt3/detection_loss‚3.581738268124296 2loss/SRqqt3/assignment_loss‚3.7326901348269717ÉÄ›ßYƒÓ 2loss/SRqqt2/assignment_loss‚3.1437744250664346 .loss/FRt4/detection_loss‚3.946899793868841 .loss/FBt1/detection_loss‚34.13238373398781 .loss/FBt2/detection_loss‚4.450126463497007 /loss/FBt4/detection_loss‚1.2710481541877836 2loss/SRqqt1/assignment_loss‚3.4306601773608816 0loss/FBt1/assignment_loss‚0.9014526844024658 1loss/FBt2/assignment_loss‚0.34578847647267014 0loss/FBt4/assignment_loss‚0.6528771820069268 0loss/SRqqt4/detection_loss‚3.209837111283322 /loss/FRt1/assignment_loss‚7.255180707344642 0loss/FRt3/assignment_loss‚7.4015690722364065 2loss/SRqqt4/assignment_loss‚3.6500508104051863 1loss/SRqqt1/detection_loss‚3.6235787238587034 _step‚19 % _timestamp‚1.7884050840041738e+09 /loss/FRt4/assignment_loss‚7.338856497476267 .loss/FRt1/detection_loss‚3.232156440615654 .loss/FRt3/detection_loss‚3.581738268124296 2loss/SRqqt3/assignment_loss‚3.7326901348269717 0loss/FBt3/assignment_loss‚0.5904110497639825 1loss/SRqqt2/detection_loss‚1.9510471010437378 /loss/FBt3/detection_loss‚5.4733860721190775 &loss/total_loss‚114.48791023151881 epoch‚0 /loss/FRt2/detection_loss‚2.5865663781762125 _runtime‚ 186.991532427 /loss/FRt2/assignment_loss‚7.027908885478974 1loss/SRqqt3/detection_loss‚1.5579218879145404 trainer/global_step‚2493¹ýB’„‚(:‡ äÊãÔ¥¡‰ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þúÐI…‚(:¾ äÊãÔÍú‡  disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1371.51416015625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.806640625 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.drbd1716.in‚0 disk.dm-0.in‚ 4.16796875 disk.sda.out‚ 10.765625# proc.memory.rssMB‚ 2705.90234375* disk./.usagePercent‚15.938718288653941 disk.drbd2080.in‚0% disk.nvme0n1.out‚1371.51416015625 disk.drbd2265.in‚0 proc.cpu.threads‚17 network.recv‚158950 disk.sda2.in‚0 disk.drbd1716.out‚0" disk.dm-0.out‚1371.51416015625 disk.dm-1.out‚ 10.765625( proc.memory.availableMB‚ 363.05859375% disk./.usageGB‚237.49842071533203 disk.drbd2266.out‚0* proc.memory.percent‚16.515517234802246 cpu‚25.129174316130964 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 10.765625$ memory_percent‚97.78406620025635 disk.nvme0n1.in‚ 4.16796875 disk.dm-1.in‚0 disk.drbd2265.out‚0 network.sent‚343985 disk.drbd2080.out‚0 disk.sda.in‚0åþ`6’†‚(:‡ óÊãÔ̱­ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.536' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o+ü L‡‚(:Á óÊãÔŒíÜ  disk.drbd2266.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.16796875 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.16796875 disk.dm-1.out‚ 11.50390625 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.sda.in‚0 disk.sda3.out‚ 11.50390625# memory_percent‚97.7837324142456 disk.sda2.out‚0& disk.nvme0c0n1.out‚1375.0478515625 disk.rbd1.out‚ 0.8388671875$ disk.nvme0n1.out‚1375.0478515625 disk.drbd2265.out‚0* proc.memory.percent‚16.515517234802246 disk.rbd0.out‚0 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2705.90234375 disk.sda3.in‚0 disk.sda2.in‚0 network.recv‚164722% disk./.usageGB‚237.49834823608398 disk.dm-1.in‚0 disk.sda.out‚ 11.50390625 cpu‚25.123353401343827 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚1375.0478515625 disk.drbd2265.in‚0( proc.memory.availableMB‚ 363.11328125 network.sent‚394048* disk./.usagePercent‚15.938713424510793Í•…0ˆ+ _wandbruntime‚223  _runtime‚223ì>#Ñĉ¾ & lr-AdamW/pg1‚6.802535640637737e-05 & lr-AdamW/pg2‚6.802535640637737e-05 trainer/global_step‚299 % _timestamp‚1.7884051200869167e+09 _runtime‚ 223.074382835 _step‚20úX3IÈŠÂ & lr-AdamW/pg1‚6.802535640637737e-05 & lr-AdamW/pg2‚6.802535640637737e-05 trainer/global_step‚299 % _timestamp‚1.7884051200869167e+09 _runtime‚ 223.074382835 _step‚20b>0‹+ _wandbruntime‚223  _runtime‚223Ù™¢ä«'Œ¥O D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9763854 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18773521393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 223.07532912 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9782396 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18515876610560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 244759947698 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16922806137344 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 245664293888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚890 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1377329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4212396 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5569998 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2456102 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17169144029696 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884051200876424e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1376886 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚871 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16486922556416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17168470431232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5569998 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3832986 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456100 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17168470431232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16486939595776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚299 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4212396 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18773716016128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16922824140288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4212398 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16922824140288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4212398 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 244759947698 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16922806137344 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 245664293888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚890 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1377329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4212396 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5569998 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2456102 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 ?9DeviceStatsMonitor.on_train_batch_®ƒA½start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17169144029696 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884051200876424e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1376886 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚871 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16486922556416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17168470431232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5569998 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3832986 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456100 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17168470431232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16486939595776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚299 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4212396 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18773716016128 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3845733 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1381465 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18577527348224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 246483721728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18577503408128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16788074539414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 223.763834007 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16979177179136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚22 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5588478 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18836031311872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4226412 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16979159176192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18836219629568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 246230813078 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3846181 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 246483721728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2464268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16541843726336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 188317696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9814892 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4226412 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 164377600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 247139330560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4226414 % _timestamp‚1.7884051207760186e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4226414 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9796324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5569912 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 258527903744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9796324 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2Ðüñõº&´M H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5569912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2464270 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258692281344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17225642897920 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3845733 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1381465 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18577527348224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 246483721728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18577503408128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16788074539414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 223.763834007 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16979177179136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚22 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5588478 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18836031311872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4226412 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16979159176192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18836219629568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 246230813078 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3846181 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 246483721728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2464268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16541843726336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 188317696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9814892 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4226412 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 164377600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 247139330560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4226414 % _timestamp‚1.7884051207760186e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4226414 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9796324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5569912 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 258527903744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9796324 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884051564367306e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11390706 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4472542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚871 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19518395841090 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 259.424407867 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20027108179456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1608040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496050 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19740476082688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19740458079744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11409248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4472069 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 300649562112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21899039115776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496050 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4913196 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 286249717314€š¹´'Ÿ®O RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 259.424407867 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20027108179456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1608040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496050 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19740476082688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19740458079744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11409248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4472069 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 300649562112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21899039115776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496050 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4913196 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 286249717314 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚349 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19517724371522 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 286650099712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20027108179456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19740458079744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2864500 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 287305695232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19232146123776 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4913198 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20027781777920 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4913198 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21598389553664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19232129084416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4913196 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚25 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6477510 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884051564367306e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11390706 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4472542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚871 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19518395841090 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477à**0 + _wandbruntime‚260  _runtime‚260åOíº&¡´M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19796829121536 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 287469527552 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4484810 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20084954257920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20084954257920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4485275 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19796829121536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19573444937046 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19287033214976 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 287066152486 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚ 260.103824194 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2872670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19574116406822 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20084280646144 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 288125136384 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4927212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚26 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6495964 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21961738957312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1612605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21660040291328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19287050254336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6495964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21660064231424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301508251136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4927212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 287469527552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2872668 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19796811118592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11423176 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301674725888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19796811118592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4927214 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚349 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21961548542464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚07¨\«g¢¸M >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1612605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21660040291328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19287050254336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6495964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21660064231424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301508251136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4927212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 287469527552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2872668 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19796811118592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11423176 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301674725888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19796811118592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4927214 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚349 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21961548542464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19796829121536 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 287469527552 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4484810 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20084954257920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20084954257920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4485275 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19796829121536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19573444937046 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19287033214976 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 287066152486 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚ 260.103824194 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2872670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19574116406822 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20084280646144 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 288125136384 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4927212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚26 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6495964 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21961738957312¬ý¡‘0£+ _wandbruntime‚260  _runtime‚260Ë?•O¤É /loss/FRt4/assignment_loss‚7.246910439597236 1loss/FBt1/assignment_loss‚0.34236734093759547 1loss/FBt3/assignment_loss‚0.48958096297186177 /loss/FRt2/detection_loss‚3.2901963784738824 1loss/SRqqt3/detection_loss‚2.4256845784091583 .loss/FBt3/detection_loss‚6.683272456750274 %loss/total_loss‚99.77467838400915 1loss/SRqqt1/detection_loss‚2.9018037660555405 /loss/FBt1/detection_loss‚14.997037269852378 .loss/FRt3/detection_loss‚3.716473328678504 $ _timestamp‚1.788405157117697e+09 _runtime‚ 260.10506315 /loss/FRt1/assignment_loss‚7.205834688142288 2loss/SRqqt4/assignment_loss‚3.5492841005325317 1loss/SRqqt2/detection_loss‚2.0285432333347657 _step‚27 2loss/SRqqt2/assignment_loss‚3.2320806333574197 .loss/FRt1/detection_loss‚3.705401977134305 -loss/FRt3/assignment_loss‚7.3340773789779 1loss/SRqqt3/assignment_loss‚3.663829333073384 0loss/FBt2/assignment_loss‚0.0681104905358175 /loss/FBt4/assignment_loss‚0.541797426632709 trainer/global_step‚349 /loss/FRt2/assignment_loss‚6.986948613767271 2loss/SRqqt1/assignment_loss‚3.3361767508766866 1loss/SRqqt4/detection_loss‚3.8636401064693926 epoch‚0 .loss/FRt4/detection_loss‚4.629887320101261 /loss/FBt2/detection_loss‚3.4317700210504416 .loss/FBt4/detection_loss‚4.103969788296538XzˆwS¥Í 1loss/SRqqt2/detection_loss‚2.0285432333347657 _step‚27 2loss/SRqqt2/assignment_loss‚3.2320806333574197 .loss/FRt1/detection_loss‚3.705401977134305 -loss/FRt3/assignment_loss‚7.3340773789779 1loss/SRqqt3/assignment_loss‚3.663829333073384 0loss/FBt2/assignment_loss‚0.0681104905358175 /loss/FBt4/assignment_loss‚0.541797426632709 trainer/global_step‚349 /loss/FRt2/assignment_loss‚6.986948613767271 2loss/SRqqt1/assignment_loss‚3.3361767508766866 1loss/SRqqt4/detection_loss‚3.8636401064693926 epoch‚0 .loss/FRt4/detection_loss‚4.629887320101261 /loss/FBt2/detection_loss‚3.4317700210504416 .loss/FBt4/detection_loss‚4.103969788296538 /loss/FRt4/assignment_loss‚7.246910439597236 1loss/FBt1/assignment_loss‚0.34236734093759547 1loss/FBt3/assignment_loss‚0.48958096297186177 /loss/FRt2/detection_loss‚3.2901963784738824 1loss/SRqqt3/detection_loss‚2.4256845784091583 .loss/FBt3/detection_loss‚6.683272456750274 %loss/total_loss‚99.77467838400915 1loss/SRqqt1/detection_loss‚2.9018037660555405 /loss/FBt1/detection_loss‚14.997037269852378 .loss/FRt3/detection_loss‚3.716473328678504 $ _timestamp‚1.788405157117697e+09 _runtime‚ 260.10506315 /loss/FRt1/assignment_loss‚7.205834688142288 2loss/SRqqt4/assignment_loss‚3.54928410053253179硱’¦‚(:‡ ¯ËãÔÀ—• gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.174' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.4348 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ×gG§‚(:¼ ¯ËãÔüÞ»  disk.dm-1.out‚ 14.5390625' disk.nvme0c0n1.out‚1377.54931640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 1.08984375 disk.sda2.out‚0$ proc.memory.availableMB‚239.1875* proc.memory.percent‚16.515564918518066 network.sent‚463315" disk.dm-0.out‚1377.54931640625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2080.in‚0% disk./.usageGB‚236.17208862304688% disk.nvme0n1.out‚1377.54931640625 disk.dm-0.in‚ 4.16796875 disk.sda.out‚ 14.5390625 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0$ memory_percent‚98.54011535644531 disk.drbd2265.in‚0 disk.sda3.out‚ 14.5390625 disk.nvme0n1.in‚ 4.16796875 disk.dm-1.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 cpu‚25.107634566949432 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 4.16796875# proc.memory.rssMB‚ 2705.91015625 proc.cpu.threads‚17 network.recv‚185216* disk./.usagePercent‚15.849707029073937¬oHø’¨‚(:‡ ¾ËãÔ¨ñû gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚127.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5056 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0  '‰J©‚(:¿ ¾ËãÔáîô  disk.sda2.in‚0 disk.drbd1716.in‚0% disk.nvme0n1.out‚1377.74462890625 disk.sda.in‚0 cpu‚25.104770492492595 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.16796875 disk.sda3.out‚ 15.26953125# memory_percent‚98.5386848449707 network.sent‚513311* disk./.usagePercent‚15.849694996719826* proc.memory.percent‚16.515612602233887 disk.drbd1716.out‚0 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚1377.74462890625# proc.memory.rssMB‚ 2705.91796875 proc.cpu.threads‚17$ disk./.usageGB‚236.1719093322754 disk.sda2.out‚0' disk.nvme0c0n1.out‚1377.74462890625 disk.drbd2265.in‚0 network.recv‚191070 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.16796875 disk.drbd2266.in‚0& proc.memory.availableMB‚ 239.421875 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.rbd1.out‚ 1.18359375 disk.dm-1.in‚0 disk.dm-1.out‚ 15.26953125 disk.drbd2265.out‚0 disk.sda.out‚ 15.26953125«cJŠ0ª+ _wandbruntime‚296  _runtime‚296¦Ä«¾ & lr-AdamW/pg1‚9.077631172623601e-05 & lr-AdamW/pg2‚9.077631172623601e-05 trainer/global_step‚399 % _timestamp‚1.7884051930164008e+09 _runtime‚ 296.003913743 _step‚28cQÒÍȬ & lr-AdamW/pg1‚9.077631172623601e-05 & lr-AdamW/pg2‚9.077631172623601e-05 trainer/global_step‚399 % _timestamp‚1.7884051930164008e+09 _runtime‚ 296.003913743 _step‚28ÑUTO0­+ _wandbruntime‚296  _runtime‚296ž1ºÙ¯'®©O K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5613996 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7422102 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22886419526144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 328291501056 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7403562 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 $ _timestamp‚1.788405193017107e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22558128025088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200913920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176973824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22885745927680 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22558128025088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7403562 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3272900 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 344004445696 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22304437708498 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22558110022144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22558110022144 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22885745927680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24680926436864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 327635905536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 343827471872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5613998 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7422102 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21977352651776 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 327085056722 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22886419526144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24680950376960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚29 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3272902 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 328291501056 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5112188 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22303766238930 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 327635905536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5613998 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1839764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5112666 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚399 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25024954822656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5613996 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21977335612416 _runtime‚ 296.004709464RêE[³'¯­O TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22558110022144 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22885745927680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24680926436864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 327635905536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 343827471872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5613998 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7422102 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21977352651776 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 327085056722 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22886419526144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24680950376960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚29 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3272902 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 328291501056 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5112188 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22303766238930 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 327635905536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5613998 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1839764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5112666 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚399 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25024954822656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5613996 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21977335612416 _runtime‚ 296.004709464 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5613996 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7422102 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22886419526144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 328291501056 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7403562 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 $ _timestamp‚1.788405193017107e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22558128025088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200913920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176973824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22885745927680 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22558128025088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7403562 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3272900 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 344004445696 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22304437708498 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22558110022144Þ‹·„0°+ _wandbruntime‚296  _runtime‚296'×4×4±³M OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_eÇ÷t7…!nd/allocation.small_pool.allocated‚7440582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13068596 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5125390 _step‚30 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚399 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24742601114624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22614481063936 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22359486804454 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22943592006144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22614481063936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788405193713535e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7440582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22614463060992 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 327247061478 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 344852936704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22032256782336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5124932 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22360158274230 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7422016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1844320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5628012 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3281068 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 344686461952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25087454051328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5628012 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25087263636480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416NÐy´½&²·M OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7440582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13068596 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5125390 _step‚30 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚399 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24742601114624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22614481063936 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22359486804454 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22943592006144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22614481063936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788405193713535e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7440582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22614463060992 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 327247061478 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 344852936704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22032256782336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5124932 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22360158274230 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7422016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1844320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5628012 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3281068 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 344686461952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25087454051328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5628012 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25087263636480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416q–DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 trainer/global_step‚449 _step‚33 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 369277306880 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14662952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 387085972480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25375779967488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 368621711360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5752830 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25090479575906 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6314798 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5753282 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25745057274368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8329614 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8348154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2071980 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24722559179776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6314796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 386932067328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24722542140416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 332.46994277 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6314796 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25745057274368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25375761964544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8329614 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 367265965922 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788405229482323e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25744383675904 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 368621711360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3681302 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28150395387392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27763463320064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25744383675904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 369277306880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8348154 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25745057274368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8329614 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8348154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2071980 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24722559179776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6314796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 386932067328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24722542140416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 332.46994277 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6314796 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25745057274368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25375761964544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8329614 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 367265965922 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788405229482323e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25744383675904 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 368621711360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3681302 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28150395387392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27763463320064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25744383675904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 369277306880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8348154 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 trainer/global_step‚449 _step‚33 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 369277306880 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14662952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 387085972480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25375779967488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 368621711360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5752830 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25090479575906 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6314798 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5753282!W¿Ì0Â+ _wandbruntime‚333  _runtime‚333q:{º&ôM MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8366634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3689470 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25146200141638 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14676880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25802229754368 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6328812 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14695448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2076487 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25432133006336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3689468 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6328814 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8348068 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5765498 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8348068 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5765957 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 368082400886 _runtime‚ 333.164032317 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚34 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 368736831302 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8366634 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27825114057728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27825137997824 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 387955010048 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25145528671862 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6328814 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28213093007872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 387788535296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24777446270976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 370096748032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 trainer/global_step‚449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚25801556142592 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28212902593024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25432115003392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24777463310336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6328812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 369441139200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928¨TJ„Ÿ"ĸM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14676880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25802229754368 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6328812 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14695448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2076487 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25432133006336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3689468 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6328814 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8348068 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5765498 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8348068 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5765957 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 368082400886 _runtime‚ 333.164032317 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚34 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 368736831302 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8366634 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27825114057728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27825137997824 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 387955010048 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25145528671862 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6328814 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28213093007872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 387788535296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24777446270976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 370096748032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 trainer/global_step‚449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚25801556142592 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28212902593024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25432115003392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24777463310336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6328812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 369441139200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8366634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3689470 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25146200141638 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0"-,¡ 0Å+ _wandbruntime‚333  _runtime‚333kLV_RÆÌ /loss/SRqqt4/detection_loss‚3.05717255289738 /loss/FBt2/detection_loss‚2.7423345797263514 _runtime‚ 333.165419314 /loss/FRt1/assignment_loss‚7.022544392084671 1loss/SRqqt1/detection_loss‚3.4031390392262004 2loss/SRqqt2/assignment_loss‚3.1930506635219493 0loss/FBt1/assignment_loss‚0.5079864208237268 .loss/FRt4/detection_loss‚4.244335202010054 /loss/FRt4/assignment_loss‚7.050308290280793 1loss/SRqqt2/detection_loss‚3.1023868664336254 .loss/FBt1/detection_loss‚8.820896201663547 .loss/FBt3/detection_loss‚6.648329383383195 epoch‚0 2loss/SRqqt1/assignment_loss‚3.3600426497666733 /loss/FRt3/assignment_loss‚6.335406020835594 0loss/FBt3/assignment_loss‚0.3433319802764648 %loss/total_loss‚88.73851746911686 trainer/global_step‚449 1loss/FBt2/assignment_loss‚0.28409429518387697 0loss/FBt4/assignment_loss‚0.8182536179374438 /loss/FRt1/detection_loss‚2.7970691042936453 /loss/FBt4/detection_loss‚3.9700389842171964 % _timestamp‚1.7884052301777685e+09 _step‚35 2loss/SRqqt3/assignment_loss‚3.2075748004411397 .loss/FRt2/detection_loss‚3.074047513306141 /loss/FRt2/assignment_loss‚7.219327732920647 2loss/SRqqt4/assignment_loss‚2.3308435632632327 /loss/FRt3/detection_loss‚2.9667812000822136 0loss/SRqqt3/detection_loss‚2.239222414541093yÛjVVÇÐ _runtime‚ 333.165419314 /loss/FRt1/assignment_loss‚7.022544392084671 1loss/SRqqt1/detection_loss‚3.4031390392262004 2loss/SRqqt2/assignment_loss‚3.1930506635219493 0loss/FBt1/assignment_loss‚0.5079864208237268 .loss/FRt4/detection_loss‚4.244335202010054 /loss/FRt4/assignment_loss‚7.050308290280793 1loss/SRqqt2/detection_loss‚3.1023868664336254 .loss/FBt1/detection_loss‚8.820896201663547 .loss/FBt3/detection_loss‚6.648329383383195 epoch‚0 2loss/SRqqt1/assignment_loss‚3.3600426497666733 /loss/FRt3/assignment_loss‚6.335406020835594 0loss/FBt3/assignment_loss‚0.3433319802764648 %loss/total_loss‚88.73851746911686 trainer/global_step‚449 1loss/FBt2/assignment_loss‚0.28409429518387697 0loss/FBt4/assignment_loss‚0.8182536179374438 /loss/FRt1/detection_loss‚2.7970691042936453 /loss/FBt4/detection_loss‚3.9700389842171964 % _timestamp‚1.7884052301777685e+09 _step‚35 2loss/SRqqt3/assignment_loss‚3.2075748004411397 .loss/FRt2/detection_loss‚3.074047513306141 /loss/FRt2/assignment_loss‚7.219327732920647 2loss/SRqqt4/assignment_loss‚2.3308435632632327 /loss/FRt3/detection_loss‚2.9667812000822136 0loss/SRqqt3/detection_loss‚2.239222414541093 /loss/SRqqt4/detection_loss‚3.05717255289738 /loss/FBt2/detection_loss‚2.7423345797263514#S„ ’È‚(:‡ úËãÔ¡Þ‚ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚157.463' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«uX<É‚(:± úËãÔ†Û¤ $ disk./.usageGB‚236.1719093322754 disk.sda1.out‚0 disk.sda.in‚0! proc.memory.rssMB‚ 2705.953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.16796875" disk.nvme0n1.out‚ 1381.44921875 disk.sda3.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 17.5390625 disk.drbd2265.out‚0 disk.sda.out‚ 17.5390625 cpu‚25.095020897498546* disk./.usagePercent‚15.849694996719826 disk.nvme0c0n1.in‚ 4.16796875 disk.dm-0.out‚ 1381.44921875 disk.dm-1.in‚0$ memory_percent‚98.54145050048828% proc.memory.availableMB‚ 238.96875 proc.cpu.threads‚17 network.sent‚581089 disk.sda3.out‚ 17.5390625 disk.drbd2266.out‚0 disk.dm-0.in‚ 4.16796875 network.recv‚209962 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 1.48828125$ disk.nvme0c0n1.out‚ 1381.44921875 disk.drbd2265.in‚0* proc.memory.percent‚16.515827178955078Í ‘Ê’Ê‚(:‡ ‰ÌãÔú•ê gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c HË‚(:½ ‰ÌãÔé‚  disk.drbd2266.out‚0 disk.nvme0n1.in‚ 4.16796875! disk.dm-0.out‚1382.1259765625 disk.dm-1.out‚ 18.05859375 disk.sda.in‚0 disk.sda.out‚ 18.05859375 disk.drbd2080.out‚0* proc.memory.percent‚16.515827178955078 cpu‚25.093303526032926 disk.rbd1.in‚0 disk.rbd1.out‚ 1.55078125 disk.dm-0.in‚ 4.16796875 disk.drbd2265.out‚0 network.recv‚215847$ disk./.usageGB‚236.1718521118164 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 4.16796875( proc.memory.availableMB‚ 239.16796875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚631360 disk.drbd1716.out‚0 disk.drbd2265.in‚0* disk./.usagePercent‚15.849691156606813 disk.sda3.in‚0 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚1382.1259765625$ disk.nvme0n1.out‚1382.1259765625$ memory_percent‚98.54023456573486! proc.memory.rssMB‚ 2705.953125 disk.sda3.out‚ 18.05859375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0*ö0Ì+ _wandbruntime‚368  _runtime‚368·ÏIÆÍÀ ' lr-AdamW/pg1‚0.00011352726704609467 ' lr-AdamW/pg2‚0.00011352726704609467 trainer/global_step‚499 % _timestamp‚1.7884052659556983e+09 _runtime‚ 368.943125164 _step‚36“T&ÊÎÄ trainer/global_step‚499 % _timestamp‚1.7884052659556983e+09 _runtime‚ 368.943125164 _step‚36 ' lr-AdamW/pg1‚0.00011352726704609467 ' lr-AdamW/pg2‚0.00011352726704609467$³ ¡M0Ï+ _wandbruntime‚368  _runtime‚368Á½’°'ЪO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9274206 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4089700 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2302916 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7015598 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27876521443314 trainer/global_step‚499 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28603021424128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 408755735538 % _timestamp‚1.7884052659563766e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2302479 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28603695022592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16289804 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31276304737280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7015596 _runtime‚ 368.943959648 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7015598 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30846024143360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 409607517184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27467748668416 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28603695022592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 _step‚37 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28193413906944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 410263112704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 409607517184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6392179 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16271262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 410263112704 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 430280593920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28603021424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844ù ŒÖ´'Ñ®O LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 409607517184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27467748668416 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28603695022592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 _step‚37 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28193413906944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 410263112704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 409607517184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6392179 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16271262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 410263112704 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 430280593920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28603021424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9274206 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4089700 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2302916 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7015598 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27876521443314 trainer/global_step‚499 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28603021424128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 408755735538 % _timestamp‚1.7884052659563766e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2302479 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28603695022592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16289804 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31276304737280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7015596 _runtime‚ 368.943959648 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7015598 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30846024143360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0%re§&0Ò+ _wandbruntime‚369  _runtime‚369 ÀmÓ´M VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16303732 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7029614 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27932242009046 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7029612 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6405395 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30907650940928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9292686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚38 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6404954 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2307086 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4097868 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9274120 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4097870 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 431155478528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27931570539270 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 411082553856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 369.632000587 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.fẗ\Mreed‚7029612 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 430997392384 trainer/global_step‚499 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16303732 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7029614 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9274120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 409572170710 % _timestamp‚1.7884052666440384e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ã™V¾&Ô¸M UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2307525 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31338830359552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16303732 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7029614 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27932242009046 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7029612 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6405395 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30907650940928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9292686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚38 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6404954 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2307086 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4097868 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9274120 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4097870 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 431155478528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27931570539270 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 411082553856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 369.632000587 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7029612 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 430997392384 trainer/global_step‚499 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16303732 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7029614 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9274120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 409572170710 % _timestamp‚1.7884052666440384e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 ODeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7716396 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33928561026560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4498102 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7032926 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4498100 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10200258 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34402037870592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33928537086464 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30212955196416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31461659172352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31462332770816 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451248918528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31462332770816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31011083852288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31461659172352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31011083852288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚41 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 450593323008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes“\9p.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952Rÿ¸'ã²O LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 449591074946 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884053023582876e+09 _runtime‚ 405.345931685 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 450593323008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30661891841154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31011065849344 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7032461 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10181718 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30212972235776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚549 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30212955196416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31461659172352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31462332770816 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451248918528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31462332770816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31011083852288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31461659172352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31011083852288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚41 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 450593323008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 473322938880 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 451248918528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2534361 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 448936644738 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7716396 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33928561026560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4498102 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7032926 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4498100 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10200258 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34402037870592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33928537086464 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34464384781824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33990187824128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 452068359680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2539389 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 474196957696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884053030504484e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31067436891136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30267859326976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10200172 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33990211764224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31519505250816 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7045659 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10200172 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7730414 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4506268 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 449753079702 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7730412 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31067418888192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4506270 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17930584 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7730414 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7045194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177831936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30267876366336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10218738 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2538926 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 450407510118 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚42 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31519505250816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30717612406678 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 452068359680N xñÂ&æ¼M K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7730412 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31067418888192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4506270 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17930584 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7730414 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7045194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177831936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30267876366336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10218738 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2538926 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 450407510118 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚42 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31519505250816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30717612406678 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 452068359680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10218738 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 trainer/global_step‚549 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 451412750848 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34464384781824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33990187824128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 452068359680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2539389 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 474196957696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884053030504484e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31067436891136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30267859326976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10200172 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33990211764224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31519505250816 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7045659 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10200172 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7730414 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4506268 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 449753079702 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 F@,Gí‚(:¼ ÔÌãÔ¬®›  network.sent‚749115 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.dm-0.in‚ 4.16796875 disk.dm-1.out‚ 21.16015625 network.recv‚240566 disk.sda2.in‚0 disk.sda3.in‚0) proc.memory.percent‚16.51589870452881% disk./.usageGB‚236.17214584350586 disk.drbd2266.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚1384.849609375 disk.drbd2265.out‚0 disk.sda.out‚ 21.16015625) disk./.usagePercent‚15.84971086918695 disk.dm-0.out‚1384.849609375 disk.sda3.out‚ 21.16015625$ memory_percent‚98.54292869567871 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚1384.849609375 disk.drbd1716.in‚0 disk.rbd1.out‚ 1.8876953125 disk.dm-1.in‚0' proc.memory.availableMB‚ 238.7265625 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2705.96484375 cpu‚25.086221519545536 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 4.16796875 disk.drbd2265.in‚0 disk.sda.in‚0E$o0î+ _wandbruntime‚442  _runtime‚442‡þ€ÆÄï¾ _step‚44 & lr-AdamW/pg1‚0.0001362782223659533 & lr-AdamW/pg2‚0.0001362782223659533 trainer/global_step‚599 % _timestamp‚1.7884053390659854e+09 _runtime‚ 442.053437819¾²`Èð % _timestamp‚1.7884053390659854e+09 _runtime‚ 442.053437819 _step‚44 & lr-AdamW/pg1‚0.0001362782223659533 & lr-AdamW/pg2‚0.0001362782223659533 trainer/global_step‚599,€ìv0ñ+ _wandbruntime‚442  _runtime‚442äΗn³'ò­O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37528058928640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8417196 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11107770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11107770 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8417198 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37011073969664 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33447933708562 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 492234724352 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7673462 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32958161724416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33828717791744 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906500 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19524966 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7673003 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19543508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8417196 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 490426414354 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 489771984146 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8417198 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33828735794688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚599 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 491579128832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34320970519040 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34320970519040 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34320296920576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33828735794688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 516961018880 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚45 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34320296920576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37527889472000 _runtime‚ 442.054346503 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4906502 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 491579128832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2766960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37011097909760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2766503 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11126310 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 492234724352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0î¶Í“·'ó±O ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 492234724352 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7673462 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32958161724416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33828717791744 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906500 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19524966 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7673003 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19543508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8417196 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 490426414354 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 489771984146 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8417198 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33828735794688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚599 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 491579128832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34320970519040 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34320970519040 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34320296920576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33828735794688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 516961018880 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚45 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34320296920576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37527889472000 _runtime‚ 442.054346503 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4906502 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 491579128832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2766960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37011097909760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2766503 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11126310 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 492234724352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37528058928640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8417196 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11107770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11107770 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8417198 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37011073969664 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33447933708562 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424-0Ìþa0ô+ _wandbruntime‚442  _runtime‚442õ­êN†!õ¸M J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11126224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2771105 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2771577 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33504325743862 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚ 442.736254179 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11144790 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37072724707328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33013082894336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8431212 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33885088833536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 493054165504 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4914670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19576004 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 517701869056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33503654274086 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34378142999040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33885070830592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8431214 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 492398556672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7686247 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37590426576384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34377469387264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚46 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37590602311168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7685773 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 491242849526 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8431214 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 P@DeviceStatsMonitor.on_train_batch_end/>œ8reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884053397479632e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19557436 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37590602311168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7685773 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 491242849526 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8431214 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884053397479632e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19557436 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2771577 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33504325743862 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚ 442.736254179 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11144790 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37072724707328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33013082894336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8431212 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33885088833536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 493054165504 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4914670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19576004 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 517701869056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33503654274086 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34378142999040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33885070830592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8431214 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 492398556672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7686247 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37590426576384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34377469387264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚46 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36233975575970 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9117998 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12052362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 559928854016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35703385291776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36646387737088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9117996 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40093610852864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 532564934656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5314900 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚49 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40093634792960 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37179608267264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2997618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9117998 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12033822 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37178934668800 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 533220530176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 531261753762 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37179608267264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21170360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36646387737088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9117996 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 533220530176 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2997161 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8312061 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40653539706880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36646369734144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0d)†ƒ²O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚649 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36646369734144 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 532564934656 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8312520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5314902 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21151818 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12033822 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚ 478.517267511 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36234647045538 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35703368252416 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 530607323554 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884053755296307e+09 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 560095342080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36233975575970 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9117998 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12052362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 559928854016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35703385291776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36646387737088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9117996 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40093610852864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 532564934656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_Nwû”2batch_start/inactive_split.large_pool.freed‚5314900 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚49 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40093634792960 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37179608267264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2997618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9117998 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12033822 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37178934668800 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 533220530176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 531261753762 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37179608267264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21170360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36646387737088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9117996 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 533220530176 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2997161 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8312061 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40653539706880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36646369734144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1631114241N ª›0„+ _wandbruntime‚479  _runtime‚479kAýe½&…·M R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37236107135488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9132014 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 560965863424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12052276 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40716073562112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35758272382976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36702740775936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36702722772992 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 534039971328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37236780747264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21184288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8324857 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 534039971328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9132012 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37236780747264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21184288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8324857 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 534039971328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9132012 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9132014 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 560965863424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12052276 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40716073562112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35758272382976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 93260349442á! 0‡+ _wandbruntime‚479  _runtime‚479K–•SˆÍ 1loss/SRqqt3/assignment_loss‚2.330559190581827 /loss/FRt2/assignment_loss‚6.913243330441988 1loss/SRqqt1/assignment_loss‚3.021365901880097 /loss/FRt3/detection_loss‚2.3967972873680052 /loss/FRt4/detection_loss‚3.6672535724937916 0loss/SRqqt2/detection_loss‚4.748444522420566 /loss/FBt4/assignment_loss‚0.431706303006245 /loss/FRt1/detection_loss‚3.7352527238073803 .loss/FBt2/detection_loss‚5.916069316912293 _runtime‚ 479.201580568 _step‚51 1loss/FBt3/assignment_loss‚0.31798727210156025 2loss/SRqqt2/assignment_loss‚3.0204976042111715 1loss/SRqqt4/detection_loss‚2.7311585185662755 /loss/FBt1/detection_loss‚10.337836105908666 %loss/total_loss‚94.81359383269529 trainer/global_step‚649 /loss/FRt3/assignment_loss‚5.615779346035373 0loss/FBt1/assignment_loss‚0.5138490450086205 0loss/FBt2/assignment_loss‚0.7917748922109604 /loss/FRt4/assignment_loss‚5.279499435424805 2loss/SRqqt4/assignment_loss‚1.9778194832351972 /loss/FRt2/detection_loss‚2.4524671114408054 .loss/FBt3/detection_loss‚1.850085003676147 epoch‚0 1loss/SRqqt1/detection_loss‚2.6241289439954256 1loss/SRqqt3/detection_loss‚2.7460366312195275 /loss/FBt4/detection_loss‚14.821835137075848 % _timestamp‚1.7884053762142076e+09 /loss/FRt1/assignment_loss‚6.572147153672718ìP"MW‰Ñ _runtime‚ 479.201580568 _step‚51 1loss/FBt3/assignment_loss‚0.31798727210156025 2loss/SRqqt2/assignment_loss‚3.0204976042111715 1loss/SRqqt4/detection_loss‚2.7311585185662755 /loss/FBt1/detection_loss‚10.337836105908666 %loss/total_loss‚94.81359383269529 trainer/global_step‚649 /loss/FRt3/assignment_loss‚5.615779346035373 0loss/FBt1/assignment_loss‚0.5138490450086205 0loss/FBt2/assignment_loss‚0.7917748922109604 /loss/FRt4/assignment_loss‚5.279499435424805 2loss/SRqqt4/assignment_loss‚1.9778194832351972 /loss/FRt2/detection_loss‚2.4524671114408054 .loss/FBt3/detection_loss‚1.850085003676147 epoch‚0 1loss/SRqqt1/detection_loss‚2.6241289439954256 1loss/SRqqt3/detection_loss‚2.7460366312195275 /loss/FBt4/detection_loss‚14.821835137075848 % _timestamp‚1.7884053762142076e+09 /loss/FRt1/assignment_loss‚6.572147153672718 1loss/SRqqt3/assignment_loss‚2.330559190581827 /loss/FRt2/assignment_loss‚6.913243330441988 1loss/SRqqt1/assignment_loss‚3.021365901880097 /loss/FRt3/detection_loss‚2.3967972873680052 /loss/FRt4/detection_loss‚3.6672535724937916 0loss/SRqqt2/detection_loss‚4.748444522420566 /loss/FBt4/assignment_loss‚0.431706303006245 /loss/FRt1/detection_loss‚3.7352527238073803 .loss/FBt2/detection_loss‚5.9160693169122933È}špŠ‚(:’ ÍãÔ¨ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.223' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.244600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iú ´K‹‚(:À ÍãÔÞ“Ë * disk./.usagePercent‚15.849718037397908 disk.sda3.out‚ 22.80078125 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 2.076171875" disk.dm-0.out‚1386.02392578125 disk.dm-1.out‚ 22.80078125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚1386.02392578125% disk.nvme0n1.out‚1386.02392578125 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.16796875$ memory_percent‚98.54474067687988 proc.cpu.threads‚17 network.recv‚255768$ disk./.usageGB‚236.1722526550293 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 238.4296875 disk.drbd2265.out‚0 disk.sda.out‚ 22.80078125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 network.sent‚811977 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.16796875) proc.memory.percent‚16.51594638824463 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 4.16796875# proc.memory.rssMB‚ 2705.97265625 cpu‚25.08266679961367]p1’Œ‚(:‡ ÍãÔǸ¶ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.017' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãl`)<‚(:± ÍãÔþ‡ë  disk.drbd2266.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 238.87109375# proc.memory.rssMB‚ 2705.97265625' disk.nvme0c0n1.out‚1386.34716796875$ memory_percent‚98.54204654693604% disk.nvme0n1.out‚1386.34716796875 disk.sda3.out‚23.25 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0) proc.memory.percent‚16.51594638824463% disk./.usageGB‚236.17231369018555 disk.sda1.in‚0 disk.dm-1.in‚0* disk./.usagePercent‚15.849722133518457 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 2.076171875" disk.dm-0.out‚1386.34716796875 disk.sda.in‚0 cpu‚25.081701000442767 disk.nvme0n1.in‚ 4.16796875 disk.dm-0.in‚ 4.16796875 disk.dm-1.out‚23.25 network.sent‚861189 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.16796875 network.recv‚260984 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda.out‚23.25 proc.cpu.threads‚179QÌ’Ž‚(:‡ ŸÍãÔ˜åö gpu.0.gpu‚35.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.923' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ÈÚìB‚(:· ŸÍãÔí’ë  network.recv‚265810 disk.sda3.out‚ 23.76171875 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0* disk./.usagePercent‚15.849722133518457 disk.rbd0.in‚0 disk.dm-0.in‚ 4.16796875 disk.dm-0.out‚ 1387.16796875 disk.dm-1.out‚ 23.76171875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 23.76171875 cpu‚25.080714613863098% disk./.usageGB‚236.17231369018555 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda.in‚0$ memory_percent‚98.54164123535156# proc.memory.rssMB‚ 2705.97265625) proc.memory.percent‚16.51594638824463 disk.sda2.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.16796875$ disk.nvme0c0n1.out‚ 1387.16796875$ proc.memory.availableMB‚238.9375 disk.rbd0.out‚0 disk.drbd1716.in‚0 network.sent‚867615 disk.rbd1.in‚0 disk.rbd1.out‚ 2.1083984375 disk.nvme0n1.in‚ 4.16796875" disk.nvme0n1.out‚ 1387.167968758a0+ _wandbruntime‚515  _runtime‚515ü¡ÌÆ‘À % _timestamp‚1.7884054121266708e+09 _runtime‚ 515.114132483 _step‚52 ' lr-AdamW/pg1‚0.00015902917768581196 ' lr-AdamW/pg2‚0.00015902917768581196 trainer/global_step‚699#mfŠV’Ä ' lr-AdamW/pg1‚0.00015902917768581196 ' lr-AdamW/pg2‚0.000159029177685811|ÜÅRt96 trainer/global_step‚699 % _timestamp‚1.7884054121266708e+09 _runtime‚ 515.114132483 _step‚524—Ú0“+ _wandbruntime‚515  _runtime‚515Ù1´'”®O PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8954099 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 573550740480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43779620863488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9818796 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39464021676544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5723300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 515.115063009 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40038246015488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12978414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39464021676544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚699 _step‚53 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12978414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230334 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38448591819776 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9818796 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39464039679488 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603449187328 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39020017443378 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40038246015488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 572097093170 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40037572417024 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40037572417024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43779440921088 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 574206336000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5723302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884054121273925e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22797212 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8953634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38448574780416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9818798 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 571442662962 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12959874 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12959874 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 574206336000 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39464039679488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9818798 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 573550740480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43779620863488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9818796 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39464021676544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5723300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 515.115063009 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40038246015488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12978414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39464021676544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚699 _step‚53 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12978414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230334 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38448591819776 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9818796 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39464039679488 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603449187328 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39020017443378 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40038246015488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 572097093170 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40037572417024 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40037572417024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43779440921088 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 574206336000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5723302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884054121273925e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22797212 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8953634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38448574780416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98187985';cÍ0–+ _wandbruntime‚515  _runtime‚515Ó´f$¾&—¸M I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43842139791872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 574370168320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9832812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604150903296 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9832814 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 572913528342 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43237822413824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9832812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 574370168320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39520374715392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39076409478678 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38503495950336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9832814 trainer/global_step‚699 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8966827 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8966350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12978328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5731470 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39075738008902 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 % _timestamp‚1.7884054128177037e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3234882 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38503478910976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22829708 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40094744883712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 575025777152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚ 515.805522139 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39520374715392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚477 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731468 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 572259097926 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12978328 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43841949377024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚54 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604317378048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22811140 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39520392718336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12996894 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0t:Pј¼M E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 572259097926 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12978328 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43841949377024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚54 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604317378048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22811140 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39520392718336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12996894 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43842139791872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 574370168320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9832812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604150903296 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9832814 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 572913528342 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43237822413824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9832812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 574370168320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39520374715392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39076409478678 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38503495950336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9832814 trainer/global_step‚699 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8966827 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8966350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12978328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5731470 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39075738008902 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 % _timestamp‚1.7884054128177037e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3234882 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38503478910976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22829708 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40094744883712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 575025777152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚ 515.805522139 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39520374715392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚477 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731468 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 1856373766÷ZWõ0™+ _wandbruntime‚515  _runtime‚515uÝà8NšÈ 1loss/SRqqt1/assignment_loss‚3.079601803763968 0loss/SRqqt1/detection_loss‚2.365887052211605 trainer/global_step‚699 .loss/FRt2/detection_loss‚2.490452607472738 .loss/FRt4/detection_loss‚3.936821992332871 0loss/SRqqt4/detection_loss‚2.466457035392523 .loss/FBt4/detection_loss‚9.297013886272907 /loss/FRt2/assignment_loss‚6.718157927195231 /loss/FBt2/assignment_loss‚0.624657352697298 /loss/FRt1/detection_loss‚3.2070255507814123 _runtime‚ 515.807010417 /loss/FBt3/detection_loss‚3.0373303169066537 epoch‚0 0loss/FRt3/assignment_loss‚6.0436794265868174 2loss/SRqqt3/assignment_loss‚1.9754448999409322 0loss/FBt1/assignment_loss‚0.3016349765560638 1loss/SRqqt3/detection_loss‚2.5184918675157757 1loss/SRqqt4/assignment_loss‚2.241162029094994 1loss/FBt3/assignment_loss‚0.12545989277979666 .loss/FRt3/detection_loss‚2.396096462295169 0loss/SRqqt2/detection_loss‚5.030939342975617 .loss/FBt1/detection_loss‚5.804507574687402 /loss/FBt2/detection_loss‚1.3838019232395409 %loss/total_loss‚79.65410654567242 $ _timestamp‚1.788405412819608e+09 /loss/FRt1/assignment_loss‚5.250118478815606 /loss/FRt4/assignment_loss‚5.881007065644136 2loss/SRqqt2/assignment_loss‚3.1138445234298704 0loss/FBt4/assignment_loss‚0.3645125570834959 _step‚55ÏUÈ#R›Ì /loss/FRt2/assignment_loss‚6.718157927195231 /loss/FBt2/assignment_loss‚0.624657352697298 /loss/FRt1/detection_loss‚3.2070255507814123 _runtime‚ 515.807010417 /loss/FBt3/detection_loss‚3.0373303169066537 epoch‚0 0loss/FRt3/assignment_loss‚6.0436794265868174 2loss/SRqqt3/assignment_loss‚1.9754448999409322 0loss/FBt1/assignment_loss‚0.3016349765560638 1loss/SRqqt3/detection_loss‚2.5184918675157757 1loss/SRqqt4/assignment_loss‚2.241162029094994 1loss/FBt3/assignment_loss‚0.12545989277979666 .loss/FRt3/detection_loss‚2.396096462295169 0loss/SRqqt2/detection_loss‚5.030939342975617 .loss/FBt1/detection_loss‚5.804507574687402 /loss/FBt2/detection_loss‚1.3838019232395409 %loss/total_loss‚79.65410654567242 $ _timestamp‚1.788405412819608e+09 /loss/FRt1/assignment_loss‚5.250118478815606 /loss/FRt4/assignment_loss‚5.881007065644136 2loss/SRqqt2/assignment_loss‚3.1138445234298704 0loss/FBt4/assignment_loss‚0.3645125570834959 _step‚55 1loss/SRqqt1/assignment_loss‚3.079601803763968 0loss/SRqqt1/detection_loss‚2.365887052211605 trainer/global_step‚699 .loss/FRt2/detection_loss‚2.490452607472738 .loss/FRt4/detection_loss‚3.936821992332871 0loss/SRqqt4/detection_loss‚2.466457035392523 .loss/FBt4/detection_loss‚9.2970138862729077†ÎÝ’œ‚(:‡ ®ÍãÔ†¡‰ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Ã€K‚(:À ®ÍãÔÖÛæ  disk.sda1.out‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 238.89453125 disk.rbd1.in‚0 network.recv‚270363 disk.rbd0.out‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚1390.8603515625! disk.dm-0.out‚1390.8603515625$ memory_percent‚98.54190349578857) proc.memory.percent‚16.51594638824463 disk.drbd2266.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.16796875 cpu‚25.079973322935288$ disk./.usageGB‚236.1754913330078 disk.sda3.out‚ 24.40234375 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚ 4.16796875$ disk.nvme0n1.out‚1390.8603515625 disk.dm-0.in‚ 4.16796875 proc.cpu.threads‚17 disk.sda3.in‚0 network.sent‚873844* disk./.usagePercent‚15.849935387794478 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 24.40234375 disk.sda.out‚ 24.40234375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 2.2021484375# proc.memory.rssMB‚ 2705.97265625ƒ…š’ž‚(:‡ ½ÍãÔòý˜ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.046' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾I ùIŸ‚(:¾ ½ÍãÔ®äô  disk.nvme0n1.in‚ 4.16796875# proc.memory.rssMB‚ 2705.97265625 disk.sda3.in‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚1392.73388671875 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 4.16796875 network.recv‚276250% disk./.usageGB‚236.17542266845703 disk.drbd2266.out‚0 disk.drbd2080.in‚0 network.sent‚924179 disk.drbd2266.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 239.140625 cpu‚25.079075914629513 disk.nvme0c0n1.in‚ 4.16796875" disk.dm-0.out‚1392.73388671875 disk.dm-1.out‚ 25.1328125 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 2.2646484375% disk.nvme0n1.out‚1392.73388671875) proc.memory.percent‚16.51594638824463) disk./.usagePercent‚15.84993077965886 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.sda.out‚ 25.1328125$ memory_percent‚98.54040145874023 disk.sda3.out‚ 25.1328125 disk.sda2.in‚0¡îÆ0 + _wandbruntime‚551  _runtime‚551¯ºÎÞÆ¡À ' lr-AdamW/pg1‚0.00017040465534574127 ' lr-AdamW/pg2‚0.00017040465534574127 trainer/global_step‚749 % _timestamp‚1.7884054487076614e+09 _runtime‚ 551.695114764 _step‚56Ň¾Ê¢Ä ' lr-AdamW/pg1‚0.00017040465534574127 ' lr-AdamW/pg2‚0.00017040465534574127 trainer/global_step‚749 % _timestamp‚1.7884054487076614e+09 _runtime‚ 551.695114764 _step‚568e}0£+ _wandbruntime‚551  _runtime‚551&‰¸'¤²O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42281691621888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6131700 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10519596 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6131702 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41806730780354 % _timestamp‚1.7884054487084181e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42896210165248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3462812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42896883763712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46905115699200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42281673618944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41193781308416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42281673618944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 612932432578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10519598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13885926 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 551.696039028 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42281691621888 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615192141824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚57 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13904466 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615192141824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3462346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9594514 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46258708559360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46258684619264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41193798347776 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13904466 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 614536546304 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42896883763712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 614536546304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 612278002370 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚749 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24424064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9594046 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42896210165248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41806059310786 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13885926 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 646587082240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10519596 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6131702 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41806730780354 % _timestamp‚1.7884054487084181e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42896210165248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3462812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42896883763712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46905115699200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42281673618944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41193781308416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42281673618944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 612932432578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10519598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13885926 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 551.696039028 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42281691621888 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615192141824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚57 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13904466 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615192141824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3462346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9594514 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46258708559360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46258684619264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41193798347776 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13904466 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 614536546304 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42896883763712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 614536546304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 612278002370 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚749 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24424064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9594046 L}Chs?DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42896210165248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41806059310786 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13885926 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 646587082240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3467480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13904380 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42953382631936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10533612 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 615355974144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13922946 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46320359297024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24456560 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 616011582976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41248685438976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 615355974144 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9606872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41862451346086 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚58 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616011582976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42338026657792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10533614 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 647455733760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 613094437334 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 160183296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 184123392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 552.379346878 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42338026657792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13922946 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24437992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3467004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24437992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3467004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3467480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13904380 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42953382631936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10533612 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 615355974144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13922946 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46320359297024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24456560 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 616011582976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41248685438976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 615355974144 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9606872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41862451346086 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚58 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616011582976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42338026657792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10533614 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 647455733760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 613094437334 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 160183296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 184123392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 552.379346878 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42338026657792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13922946:}‘âh0©+ _wandbruntime‚552  _runtime‚552>§zÄOªÉ 2loss/SRqqt3/assignment_loss‚2.2552952912190687 .loss/FRt2/detection_loss‚2.316389226913452 /loss/FRt4/detection_loss‚2.3834593121021515 .loss/FBt1/detection_loss‚5.296031669378281 trainer/global_step‚749 0loss/SRqqt2/detection_loss‚4.158001035451889 .loss/FBt2/detection_loss‚2.329682721722757 %loss/total_loss‚77.33778079059476 .loss/FRt3/detection_loss‚2.655751689736332 .loss/FBt3/detection_loss‚3.204507315265281 /loss/FRt2/assignment_loss‚6.961426575978597 /loss/FRt3/assignment_loss‚5.938393209661756 /loss/FRt4/assignment_loss‚4.553739945093791 1loss/SRqqt1/assignment_loss‚3.167843225154471 1loss/SRqqt2/assignment_loss‚3.257058867386409 1loss/FBt3/assignment_loss‚0.21553326583897442 1loss/SRqqt4/detection_loss‚1.9327174274152832 epoch‚0 1loss/FBt1/assignment_loss‚0.47294690551250823 0loss/FBt2/assignment_loss‚0.5587911149147223 0loss/FBt4/assignment_loss‚0.6326215059239075 1loss/SRqqt3/detection_loss‚2.9537941508966945 $ _timestamp‚1.788405449393106e+09 _runtime‚ 552.380471108 /loss/FRt1/assignment_loss‚7.234029080186572 2loss/SRqqt4/assignment_loss‚1.8052888477319164 .loss/FRt1/detection_loss‚2.706839805202825 0loss/SRqqt1/detection_loss‚3.039175839500224 /loss/FBt4/detection_loss‚7.3084627624068945 _step‚59ì`S«Í 2loss/SRqqt3/assignment_loss‚2.2552952912190687 .loss/FRt2/detection_loss‚2.316389226913452 /loss/FRt4/detection_loss‚2.3834593121021515 .loss/FBt1/detection_loss‚5.296031669378281 trainer/global_step‚749 0loss/SRqqt2/detection_loss‚4.158001035451889 .loss/FBt2/detection_loss‚2.329682721722757 %loss/total_loss‚77.33778079059476 .loss/FRt3/detection_loss‚2.655751689736332 .loss/FBt3/detection_loss‚3.204507315265281 /loss/FRt2/assignment_loss‚6.961426575978597 /loss/FRt3/assignment_loss‚5.938393209661756 /loss/FRt4/assignment_loss‚4.553739945093791 1loss/SRqqt1/assignment_loss‚3.167843225154471 1loss/SRqqt2/assignment_loss‚3.257058867386409 1loss/FBt3/assignment_loss‚0.21553326583897442 1loss/SRqqt4/detection_loss‚1.9327174274152832 epoch‚0 1loss/FBt1/assignment_loss‚0.47294690551250823 0loss/FBt2/assignment_loss‚0.5587911149147223 0loss/FBt4/assignment_loss‚0.6326215059239075 1loss/SRqqt3/detection_loss‚2.9537941508966945 $ _timestamp‚1.788405449393106e+09 _runtime‚ 552.380471108 /loss/FRt1/assignment_loss‚7.234029080186572 2loss/SRqqt4/assignment_loss‚1.8052888477319164 .loss/FRt1/detection_loss‚2.706839805202825 0loss/SRqqt1/detection_loss‚3.039175839500224 /loss/FBt4/detection_loss‚7.3084627624068945 _step‚59;]LØÇ’¬‚(:‡ ÌÍãÔ”àù gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.644' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3288 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 /HM­‚(: ÌÍãÔ¾±×  disk.rbd1.out‚ 2.2646484375% disk.nvme0n1.out‚1392.96044921875) proc.memory.percent‚16.51594638824463 network.recv‚280433 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 25.64453125 disk.rbd1.in‚0( proc.memory.availableMB‚ 238.41015625 network.sent‚929851* disk./.usagePercent‚15.849944348058177 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 4.16796875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0$ memory_percent‚98.54485988616943 cpu‚25.07800439836591$ disk./.usageGB‚236.1756248474121 disk.sda.out‚ 25.64453125 disk.rbd0.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚1392.96044921875 disk.nvme0n1.in‚ 4.16796875 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2705.97265625" disk.dm-0.out‚1392.96044921875 disk.dm-1.out‚ 25.64453125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.drbd2080.in‚0îq'®‚(:’ ÛÍãÔ¹–—  gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.198' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.439600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x¶EG¯‚(:¼ ÛÍãÔ‹¹É  disk.dm-1.out‚ 26.14453125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 26.14453125 disk.drbd2266.out‚0 disk.sda1.in‚0$ proc.memory.availableMB‚238.4375* disk./.usagePercent‚15.849941787982836 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0% disk./.usageGB‚236.17558670043945 disk.drbd2266.in‚0$ memory_percent‚98.54469299316406# proc.memory.rssMB‚ 2705.97265625) proc.memory.percent‚16.51594638824463 proc.cpu.threads‚17% disk.nvme0n1.out‚1393.35302734375 disk.drbd2265.out‚0 disk.sda.out‚ 26.14453125 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚2.359375 disk.dm-0.in‚ 4.16796875 disk.dm-1.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.rbd1.in‚0 network.sent‚980032 disk.nvme0n1.in‚ 4.16796875" disk.dm-0.out‚1393.35302734375 disk.sda.in‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚1393.35302734375 cpu‚25.077306949706852 network.recv‚28589409>’°‚(:‡ êÍãÔÌ—¤ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùP-RH±‚(:½ êÍãÔáóƒ $ memory_percent‚98.54319095611572# proc.memory.rssMB‚ 2705.97265625 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda.in‚0 disk.sda3.in‚0 network.sent‚985998 disk.rbd0.in‚0 disk.dm-0.in‚ 4.16796875( proc.memory.availableMB‚ 238.68359375% disk./.usageGB‚236.17564010620117 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 26.5859375 disk.drbd2080.out‚0 cpu‚25.076595861351286 disk.sda1.out‚0 disk.drbd2266.in‚0) proc.memory.percent‚16.51594638824463 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚ 4.16796875' disk.nvme0c0n1.out‚1393.56396484375% disk.nvme0n1.out‚1393.56396484375 disk.drbd2265.out‚0 disk.rbd1.out‚2.453125" disk.dm-0.out‚1393.56396484375 disk.dm-1.out‚ 26.5859375 disk.sda3.out‚ 26.5859375 disk.drbd2266.out‚0 disk.drbd2080.in‚0 network.recv‚290809* disk./.usagePercent‚15.849945372088314 disk.sda2.in‚0 disk.nvme0n1.in‚ 4.16796875¸syB0²+ _wandbruntime‚587  _runtime‚587âØÄƳÀ _step‚60 ' lr-AdamW/pg1‚0.00018178013300567062 ' lr-AdamW/pg2‚0.00018178013300567062 trainer/global_step‚799 % _timestamp‚1.7884054849962409e+09 _runtime‚ 587.983709505§[‘ñÊ´Ä % _timestamp‚1.7884054849962409e+09 _runtime‚ 587.983709505 _step‚60 ' lr-AdamW/pg1‚0.00018178013300567062 ' lr-AdamW/pg2‚0.00018178013300567062 trainer/global_step‚799<ÂEg‡0µ+ _wandbruntime‚587  _runtime‚587õJ", ¶´O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11220396 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3694845 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44592101178194 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26032374 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45099325561344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11220398 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 689571738624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 587.984657323 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14830518 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 689729838080 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43938987836416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26050916 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 182039552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49341221502464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3694379 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚799 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 655522352128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50030975280640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 653767771986 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45099325561344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10234479 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45755521511936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14811978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45099343564288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14811978 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 656177947648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49341245442560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 655522352128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6540102 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884054849970245e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43939004875776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45754847913472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44592101178194 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26032374 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45099325561344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11220398 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 689571738624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 587.984657323 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14830518 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 689729838080 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43938987836416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26050916 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 182039552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49341221502464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3694379 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚799 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 655522352128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50030975280640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 653767771986 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45099325561344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10234479 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45755521511936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14811978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45099343564288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14811978 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 656177947648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49341245442560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 655522352128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6540102 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884054849970245e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43939004875776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45754847913472=‡¹¾0¸+ _wandbruntime‚588  _runtime‚588G¸5Ã&¹½M C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26064844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179929088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44648493213494 _runtime‚ 588.672993473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 690603436544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 653929776742 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45155696603136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43993909006336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11234412 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155988992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26083412 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10247250 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43993891966976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 $ _timestamp‚1.788405485685144e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6548270 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14830432 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 654584207158 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11234414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3699443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11234414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45812693991936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11234412 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45155696603136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10247713 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3698982 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44647821743718 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928à:}枺ÁM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43993891966976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 $ _timestamp‚1.788405485685144e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6548270 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14830432 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 654584207158 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11234414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3699443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11234414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45812693991936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11234412 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45155696603136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10247713 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3698982 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44647821743718 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26064844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179929088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44648493213494 _runtime‚ 588.672993473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 `ODeviceStatsMonitor.on_trainIn–>)_batch_end/inactive_split_bytes.small_pool.allocated‚ 690603436544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 653929776742 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45155696603136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43993909006336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11234412 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155988992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26083412 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10247250 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412>(¤ 0»+ _wandbruntime‚588  _runtime‚588–­VM¼Ç 0loss/FBt1/assignment_loss‚0.6479890656852915 .loss/FRt1/detection_loss‚4.014737584374168 0loss/SRqqt1/detection_loss‚3.027022773915149 1loss/SRqqt2/detection_loss‚3.1909486800432205 % _timestamp‚1.7884054856869075e+09 %loss/total_loss‚78.16021641782918 trainer/global_step‚799 /loss/FRt1/assignment_loss‚7.226540926730994 _runtime‚ 588.674275311 1loss/SRqqt1/assignment_loss‚2.988447042221719 2loss/SRqqt3/assignment_loss‚1.8504046929734095 0loss/FBt2/assignment_loss‚0.2956981344087399 /loss/FBt3/detection_loss‚1.6027166283231054 epoch‚0 /loss/FRt4/assignment_loss‚4.754874363751479 .loss/FRt4/detection_loss‚2.100197654374888 _step‚63 /loss/FRt2/assignment_loss‚6.699780097374549 1loss/SRqqt2/assignment_loss‚2.957815903074601 0loss/FBt4/assignment_loss‚0.6331049597320089 1loss/SRqqt4/detection_loss‚1.3168593757255103 .loss/FBt1/detection_loss‚6.569393978307121 .loss/FBt4/detection_loss‚8.650611838698387 /loss/FRt3/assignment_loss‚5.567546367645264 2loss/SRqqt4/assignment_loss‚2.3745990491682485 .loss/FRt2/detection_loss‚2.433637655698336 .loss/FRt3/detection_loss‚2.027661921692566 .loss/FBt2/detection_loss‚3.240483834594488 /loss/FBt3/assignment_loss‚0.527528131033129 1loss/SRqqt3/detection_loss‚3.4616157582827976Ö‡hJQ½Ë 1loss/SRqqt2/assignment_loss‚2.957815903074601 0loss/FBt4/assignment_loss‚0.6331049597320089 1loss/SRqqt4/detection_loss‚1.3168593757255103 .loss/FBt1/detection_loss‚6.569393978307121 .loss/FBt4/detection_loss‚8.650611838698387 /loss/FRt3/assignment_loss‚5.567546367645264 2loss/SRqqt4/assignment_loss‚2.3745990491682485 .loss/FRt2/detection_loss‚2.433637655698336 .loss/FRt3/detection_loss‚2.027661921692566 .loss/FBt2/detection_loss‚3.240483834594488 /loss/FBt3/assignment_loss‚0.527528131033129 1loss/SRqqt3/detection_loss‚3.4616157582827976 0loss/FBt1/assignment_loss‚0.6479890656852915 .loss/FRt1/detection_loss‚4.014737584374168 0loss/SRqqt1/detection_loss‚3.027022773915149 1loss/SRqqt2/detection_loss‚3.1909486800432205 % _timestamp‚1.7884054856869075e+09 %loss/total_loss‚78.16021641782918 trainer/global_step‚799 /loss/FRt1/assignment_loss‚7.226540926730994 _runtime‚ 588.674275311 1loss/SRqqt1/assignment_loss‚2.988447042221719 2loss/SRqqt3/assignment_loss‚1.8504046929734095 0loss/FBt2/assignment_loss‚0.2956981344087399 /loss/FBt3/detection_loss‚1.6027166283231054 epoch‚0 /loss/FRt4/assignment_loss‚4.754874363751479 .loss/FRt4/detection_loss‚2.100197654374888 _step‚63 /loss/FRt2/assignment_loss‚6.699780097374549?òÁ•’¾‚(:‡ ùÍãÔÌä¦ gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.728' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÝe¢G¿‚(:¼ ùÍãÔÐÞ  network.sent‚992945 disk.rbd1.in‚0 disk.rbd1.out‚2.515625 disk.drbd2266.out‚0 disk.sda1.in‚0$ memory_percent‚98.54514598846436 disk.sda1.out‚0 disk.dm-0.in‚ 4.16796875" disk.dm-0.out‚1395.10205078125 disk.drbd2080.out‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2705.9765625) proc.memory.percent‚16.51597023010254 proc.cpu.threads‚17' disk.nvme0c0n1.out‚1395.10205078125( proc.memory.availableMB‚ 238.36328125 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.recv‚296081% disk.nvme0n1.out‚1395.10205078125 disk.dm-1.out‚ 27.1796875 disk.nvme0c0n1.in‚ 4.16796875* disk./.usagePercent‚15.849941787982836 disk.nvme0n1.in‚ 4.16796875 disk.sda.out‚ 27.1796875 disk.sda3.in‚0 disk.sda3.out‚ 27.1796875 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚236.17558670043945 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 cpu‚25.075982020113774ÇÎ<’À‚(:‡ ˆÎãÔØ¿ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚131.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã«Ÿ-JÁ‚(:¿ ˆÎãÔÅÚ›  disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 27.59765625 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚98.54516983032227& proc.memory.availableMB‚ 238.359375 network.sent‚ 1.0423e+06 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0% disk./.usageGB‚236.17560958862305 disk.dm-1.out‚ 27.59765625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚301360) disk./.usagePercent‚15.84994332402804" disk.dm-0.out‚1395.34033203125 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚2.515625 disk.dm-0.in‚ 4.16796875 disk.nvme0c0n1.in‚ 4.16796875' disk.nvme0c0n1.out‚1395.34033203125 disk.rbd1.in‚0" proc.memory.rssMB‚ 2705.9765625 cpu‚25.07533385757706% disk.nvme0n1.out‚1395.34033203125 disk.sda3.out‚ 27.59765625 disk.drbd2080.in‚0) proc.memory.percent‚16.51597023010254 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.16796875Î|K0Â+ _wandbruntime‚624  _runtime‚624øhbxÅÿ ' lr-AdamW/pg2‚0.00019315561066559993 trainer/global_step‚849 $ _timestamp‚1.788405521649521e+09 _runtime‚ 624.636991107 _step‚64 ' lr-AdamW/pg1‚0.00019315561066559993Nßd·ÉÄà ' lr-AdamW/pg1‚0.00019315561066559993 ' lr-AdamW/pg2‚0.00019315561066559993 trainer/global_step‚849 $ _timestamp‚1.788405521649521e+09 _runtime‚ 624.636991107 _step‚64@´¹bŽ0Å+ _wandbruntime‚624  _runtime‚624©ØÍ.º'Æ´O C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693948681186 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694603111394 % _timestamp‚1.7884055216502783e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 697163753472 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 733013143552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47378814515170 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48613485661696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47916995506688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47916977503744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11921196 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15756570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52423782325760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15738030 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3927446 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47916995506688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10875946 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6948502 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6948500 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15756570 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27659226 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11921198 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48613485661696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52423758385664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53156795469312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46684211403776 _step‚65 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 696508157952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48614159260160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 697163753472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15738030 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47378143045602 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚ 624.638037742 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 696508157952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11921198 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47916977503744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48614159260160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10876378 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 732867627008›wþ'ǸO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53156795469312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46684211403776 _step‚65 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 696508157952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48614159260160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 697163753472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15738030 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47378143045602 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚ 624.638037742 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 696508157952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11921198 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47916977503744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48614159260160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10876378 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 732867627008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693948681186 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694603111394 % _timestamp‚1.7884055216502783e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 697163753472 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 733013143552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47378814515170 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48613485661696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47916995506688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47916977503744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11921196 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15756570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52423782325760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15738030 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3927446 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47916995506688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10875946 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6948502 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6948500 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15756570 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27659226 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11921198 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48613485661696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52423758385664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0A`~AÄ0È+ _wandbruntime‚625  _runtime‚625–®NɾM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47973330542592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11935214 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0*²Èv I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48671331740160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697983194624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53219330193920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48670658128384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46739098494976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11935212 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10888735 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27710264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52485409123328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3932067 _step‚66 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 697983194624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 694765116150 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27691696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10889186 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52485433063424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47434535080902 % _timestamp‚1.7884055223368104e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48671331740160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15756484 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430pT’È&ÊÂM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11935214 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15775050 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10888735 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27710264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52485409123328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3932067 _step‚66 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 697983194624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 694765116150 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27691696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10889186 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52485433063424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47434535080902 % _timestamp‚1.7884055223368104e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48671331740160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15756484 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47973330542592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11935214 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48671331740160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697983194624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53219330193920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48670658128384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46739098494976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11935212 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ODeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12621998 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11517179 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 776090788864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16664082 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29304620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12621996 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 738149559296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 776253082624 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 735438450802 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4159801 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51472123409920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49429417931776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 734784020594 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16682622 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 737493963776 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56282572291584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56282386057728 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51472123409920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11516701 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51472123409920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49429417931776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 734784020594 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16682622 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 737493963776 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56282572291584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56282386057728 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51472123409920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11516701 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12621998 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11517179 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 776090788864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16664082 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29304620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12621996 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 738149559296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 776253082624 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 735438450802 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4159801 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 777122755072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12636014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7365070 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55567946006528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56344916966912 % _timestamp‚1.7884055585410955e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49484305022976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12636014 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16701102 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12636012 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚899 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11529525 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49484322062336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16701102 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50220576948310 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 _runtime‚ 661.528957257 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51529969488384 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4164457 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50219905478534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50790982484992 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 738313391616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55567969946624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56345092701696 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 735600455558ãpÅâF$ÜÂM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51529295876608 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4164924 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 777122755072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12636014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7365070 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55567946006528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56344916966912 % _timestamp‚1.7884055585410955e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49484305022976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12636014 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16701102 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12636012 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚899 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11529525 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49484322062336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16701102 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50220576948310 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 _runtime‚ 661.528957257 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51529969488384 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4164457 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50219905478534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50790982484992 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 738313391616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55567969946624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56345092701696 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 735600455558 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59408054146048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172779520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 776273790210 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17608674 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 819394773504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13322796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52950898249986 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52174624459776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4391682 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196719616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52950226780418 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54330761158144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17590134 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7765302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7765300 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59408250865664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54330761158144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12156485 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208·6xî¾'é¸O VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 776273790210 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17608674 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 819394773504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13322796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52950898249986 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52174624459776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4391682 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196719616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52950226780418 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54330761158144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17590134 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7765302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7765300 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59408250865664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54330761158144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12156485 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 _runtime‚ 697.126967323 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12156984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884055941393561e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778479769600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13322796 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779135365120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13322798 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59408054146048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172779520IîÄ0ê+ _wandbruntime‚697  _runtime‚697ÎÂlÎÇë¾M C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53608634427392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59470574795776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779299197440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884055948206475e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7773470 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17627154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30963968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52229528590336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53608652430336 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 779954806272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17608588 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 820249992192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30945400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13336814 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 697.808511571 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13336814 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚477 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59470756822016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53608634427392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚479 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12169198 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 820091906048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53006618815718 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 776435794966 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚74 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13336812 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12169677 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4396207 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E)È&ìÂM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53006618815718 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 776435794966 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚74 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13336812 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12169677 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4396207 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53608634427392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59470574795776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779299197440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884055948206475e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7773470 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17627154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30963968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52229528590336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53608652430336 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 779954806272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17608588 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 820249992192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30945400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13336814 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 697.808511571 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13336814 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚477 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59470756822016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53608634427392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚479 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12169198 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 820091906048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0J”)ÚÚ0í+ _wandbruntime‚697  _runtime‚697™ÚÎ3TîÎ 0loss/FRt4/assignment_loss‚4.4607333972536285 .loss/FBt1/detection_loss‚5.712504994869232 .loss/FBt2/detection_loss‚3.340622822088545 epoch‚0 1loss/SRqqt3/detection_loss‚2.2215067765047385 % _timestamp‚1.7884055948225536e+09 1loss/SRqqt2/detection_loss‚2.7596475446925446 %loss/total_loss‚69.30320030402561 _runtime‚ 697.809931169 /loss/FRt2/assignment_loss‚5.904066168743631 2loss/SRqqt2/assignment_loss‚2.8813011120347416 /loss/FRt4/detection_loss‚2.4210041196181855 1loss/SRqqt1/detection_loss‚3.6899115507091795 /loss/FBt3/detection_loss‚1.9402483676354965 .loss/FBt4/detection_loss‚5.809558749198914 /loss/FRt1/assignment_loss‚5.079695138931275 0loss/FRt3/assignment_loss‚4.1550831611339865 1loss/SRqqt1/assignment_loss‚3.411648895059313 0loss/FBt1/assignment_loss‚0.5876658236902278 0loss/FBt2/assignment_loss‚0.5814639606579644 0loss/FBt3/assignment_loss‚0.2141866683038968 0loss/FBt4/assignment_loss‚0.5630369133182934 1loss/SRqqt4/detection_loss‚1.5129832055833605 _step‚75 2loss/SRqqt3/assignment_loss‚2.0379890352487564 1loss/SRqqt4/assignment_loss‚2.641935336589813 /loss/FRt3/detection_loss‚2.2571043744683266 trainer/global_step‚949 .loss/FRt1/detection_loss‚2.507106239795685 /loss/FRt2/detection_loss‚2.6121959478958794‰¯Á=XïÒ /loss/FRt2/assignment_loss‚5.904066168743631 2loss/SRqqt2/assignment_loss‚2.8813011120347416 /loss/FRt4/detection_loss‚2.4210041196181855 1loss/SRqqt1/detection_loss‚3.6899115507091795 /loss/FBt3/detection_loss‚1.9402483676354965 .loss/FBt4/detection_loss‚5.809558749198914 /loss/FRt1/assignment_loss‚5.079695138931275 0loss/FRt3/assignment_loss‚4.1550831611339865 1loss/SRqqt1/assignment_loss‚3.411648895059313 0loss/FBt1/assignment_loss‚0.5876658236902278 0loss/FBt2/assignment_loss‚0.5814639606579644 0loss/FBt3/assignment_loss‚0.2141866683038968 0loss/FBt4/assignment_loss‚0.5630369133182934 1loss/SRqqt4/detection_loss‚1.5129832055833605 _step‚75 2loss/SRqqt3/assignment_loss‚2.0379890352487564 1loss/SRqqt4/assignment_loss‚2.641935336589813 /loss/FRt3/detection_loss‚2.2571043744683266 trainer/global_step‚949 .loss/FRt1/detection_loss‚2.507106239795685 /loss/FRt2/detection_loss‚2.6121959478958794 0loss/FRt4/assignment_loss‚4.4607333972536285 .loss/FBt1/detection_loss‚5.712504994869232 .loss/FBt2/detection_loss‚3.340622822088545 epoch‚0 1loss/SRqqt3/detection_loss‚2.2215067765047385 % _timestamp‚1.7884055948225536e+09 1loss/SRqqt2/detection_loss‚2.7596475446925446 %loss/total_loss‚69.30320030402561 _runtime‚ 697.809931169Kàx«ð‚(:’ âÎãԢĶ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.510799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—e.kIñ‚(:¾ âÎãÔÓ„À  disk.sda3.in‚0 disk.sda3.out‚ 31.8046875 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 2.916015625 proc.cpu.threads‚17! disk.dm-0.out‚1400.2470703125 disk.drbd2265.in‚0 disk.drbd2266.out‚0& disk.nvme0c0n1.out‚1400.2470703125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.16796875 network.recv‚329667 disk.dm-0.in‚ 4.16796875 disk.sda.out‚ 31.8046875 disk.nvme0c0n1.in‚ 4.16796875' proc.memory.availableMB‚ 239.0234375! proc.memory.rssMB‚ 2705.984375 cpu‚25.07116294763573) disk./.usagePercent‚15.76090953578502 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 network.sent‚ 1.165306e+06 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚98.54111671447754) proc.memory.percent‚16.51601791381836 disk.dm-1.out‚ 31.8046875 disk.sda.in‚0 disk.drbd2080.in‚0% disk./.usageGB‚234.84894180297852 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚1400.2470703125û´VŠò‚(:’ ñÎãÔ䃡 gpu.0.gpu‚12.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚131.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.262400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä1àOó‚(:Ä ñÎãÔÊÖŒ  disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.16796875$ memory_percent‚98.54185581207275 disk.rbd0.in‚0& disk.nvme0c0n1.out‚1401.0029296875 network.recv‚335323% disk./.usageGB‚234.84888458251953 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0( proc.memory.availableMB‚ 238.90234375 disk.dm-0.in‚ 4.16796875! disk.dm-0.out‚1401.0029296875 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0$ disk.nvme0n1.out‚1401.0029296875 proc.cpu.threads‚17 disk.sda3.out‚ 32.37890625* disk./.usagePercent‚15.760905695672006 disk.sda.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.51601791381836 disk.sda.out‚ 32.37890625 disk.rbd0.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2705.984375 cpu‚25.070765196791918 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 4.16796875 network.sent‚ 1.215311e+06 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 32.37890625 disk.drbd1716.in‚0 disk.rbd1.out‚ 2.978515625 disk.drbd2080.in‚0¨´K²0ô+ _wandbruntime‚733  _runtime‚733ÆI fÄõ¾ & lr-AdamW/pg2‚0.0002272820436453879 trainer/global_step‚999 % _timestamp‚1.7884056303745854e+09 _runtime‚ 733.362002155 _step‚76 & lr-AdamW/pg1‚0.0002272820436453879P>>ÃÈö & lr-AdamW/pg1‚0.0002272820436453879 & lr-AdamW/pg2‚0.0002272820436453879 trainer/global_step‚999 % _timestamp‚1.7884056303745854e+09 _runtime‚ 733.362002155 _step‚76L©À 0÷+ _wandbruntime‚733  _runtime‚733¶7¾ûº'ø´O YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62533766770176 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 862568417280 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56369951333888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57190072504832 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14023596 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 862397734912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 _step‚77 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57190072504832 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 _runtime‚ 733.362918794 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12798119 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12797649 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 817109129618 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18516186 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4624417 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54919830987776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61671369035264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55736940117394 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 816454699410 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57189398906368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32558324 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8173700 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚999 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14023596 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 820121170944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55736268647826 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18516186 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 820121170944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56369951333888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14023598 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57189398906368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56369933330944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605ÀˆJi~ù¸O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56369933330944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62533766770176 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 862568417280 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56369951333888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57190072504832 PCDeviceStatsMonitor.on_train_batch&À2@!_start/allocation.large_pool.freed‚14023596 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 862397734912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 _step‚77 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57190072504832 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 _runtime‚ 733.362918794 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12798119 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12797649 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 817109129618 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18516186 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4624417 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54919830987776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61671369035264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55736940117394 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 816454699410 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57189398906368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32558324 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8173700 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚999 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14023596 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 820121170944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55736268647826 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18516186 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 820121170944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56369951333888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14023598 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57189398906368MõÀ\70ú+ _wandbruntime‚734  _runtime‚734C2ÿÃÄ&û¾M [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62596459435520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14037612 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4628549 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12810417 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55791989213350 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 820940612096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18553206 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198803456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61733019772928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56426304372736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4629022 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚999 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57246571373056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174863360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884056310556571e+09 _runtime‚ 734.043453747 _step‚78 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 863415722496 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 863240859136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14037612 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12810892 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61733043713024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56426304372736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32572252 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18553206 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 820285003264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18534640ßÖdžÈ&üÂM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14037614 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 817925564790 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14037614 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32572252 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18553206 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 820285003264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18534640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62596459435520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14037612 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4628549 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12810417 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55791989213350 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 820940612096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18553206 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198803456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61733019772928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56426304372736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4629022 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚999 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57246571373056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174863360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884056310556571e+09 _runtime‚ 734.043453747 _step‚78 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 863415722496 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 863240859136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14037612 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12810892 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61733043713024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56426304372736 F…‚(:³ žÏãÔäÝØ  disk.nvme0n1.in‚ 4.16796875 disk.nvme0n1.out‚ 1401.828125 disk.drbd2265.out‚0 disk.sda3.out‚ 33.98828125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 33.98828125 disk.sda2.out‚0 network.sent‚ 1.27617e+06) disk./.usagePercent‚15.76092592026721% disk./.usageGB‚234.84918594360352 proc.cpu.threads‚17 disk.dm-0.out‚ 1401.828125 disk.dm-1.out‚ 33.98828125 disk.drbd2080.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.out‚ 1401.828125$ memory_percent‚98.54068756103516% proc.memory.availableMB‚ 239.09375 disk.rbd1.out‚ 3.1044921875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2705.984375 disk.dm-0.in‚ 4.16796875 disk.sda2.in‚0) proc.memory.percent‚16.51601791381836 cpu‚25.06922760739784 network.recv‚348213 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.rbd1.in‚0ÔŸVè0†+ _wandbruntime‚770  _runtime‚770õ´$ÅLJÁ trainer/global_step‚1049 % _timestamp‚1.7884056670512037e+09 _runtime‚ 770.038642012 _step‚80 ' lr-AdamW/pg1‚0.00023865752130531722 ' lr-AdamW/pg2‚0.00023865752130531722Z’^ìˈŠtrainer/global_step‚1049 % _timestamp‚1.7884056670512037e+09 _runtime‚ 770.038642012 _step‚80 ' lr-AdamW/pg1‚0.00023865752130531722 ' lr-AdamW/pg2‚0.00023865752130531722PþbÐ0‰+ _wandbruntime‚770  _runtime‚770‘ºó º'Š´O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58522310515234 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 860451381248 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 905792368128 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34166634 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59187603276288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚1049 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57665037515776 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13437854 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 905627977216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 861106976768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14724396 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60048036654592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60048710253056 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60048036654592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19442238 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4856215 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64753905918464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14724398 $ _timestamp‚1.788405667051977e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57665020476416 _runtime‚ 770.039635406 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460778 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19442238 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 861106976768 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 860451381248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8582102 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8582100 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59187603276288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0´×J·¾'‹¸O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60048710253056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60048036654592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60048710253056 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60048036654592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19442238 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4856215 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64753905918464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14724398 $ _timestamp‚1.788405667051977e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57665020476416 _runtime‚ 770.039635406 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460778 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19442238 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 861106976768 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 860451381248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8582102 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8582100 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59187603276288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58522310515234 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 860451381248 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 905792368128 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34166634 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59187603276288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚1049 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57665037515776 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13437854 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 905627977216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 861106976768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14724396 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IŽã`"¿M R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60105209121280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59243938312192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8590270 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65722052076544 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 906495420416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58578702550534 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19479258 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19479258 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65722238297088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64815580596224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚1049 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60105882733056 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚82 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34199104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59243956315136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 861926417920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14738412 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59243956315136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59243938312192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57719924606976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 861270809088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 858106473782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4860466 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14738412 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13450734 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60105882733056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13451188 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14738414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19460692 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19460692 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 858760904198 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14738414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34199104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4860918 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58578031080758 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 906657700864 % _timestamp‚1.7884056677414181e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚6481555‰ŽIce6656128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 861270809088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34217672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14738412 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59243956315136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59243938312192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57719924606976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 861270809088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 858106473782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4860466 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14738412 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13450734 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60105882733056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13451188 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14738414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19460692 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19460692 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 858760904198 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14738414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34199104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4860918 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58578031080758 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 906657700864 % _timestamp‚1.7884056677414181e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64815556656128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 861270809088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34217672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19479258 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65722238297088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64815580596224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973RÃ|0+ _wandbruntime‚770  _runtime‚770›wXÒ _step‚83 /loss/FRt2/assignment_loss‚5.829993373469303 /loss/FRt4/detection_loss‚2.4200611655358917 .loss/FBt1/detection_loss‚4.926826904217402 trainer/global_step‚1049 _runtime‚ 770.730738561 2loss/SRqqt3/assignment_loss‚2.1466389216263506 1loss/FBt2/assignment_loss‚0.48692540195939915 0loss/FBt3/assignment_loss‚0.2628579608636334 /loss/FRt2/detection_loss‚2.6976256370544434 /loss/FRt3/detection_loss‚2.2740715390190167 1loss/SRqqt3/detection_loss‚1.7009711454019827 1loss/SRqqt4/detection_loss‚1.5778165957733485 /loss/FBt2/detection_loss‚3.1522061033495543 1loss/SRqqt1/detection_loss‚2.5274138674139977 $ _timestamp‚1.788405667743369e+09 0loss/FRt3/assignment_loss‚3.3557995925558375 /loss/FRt4/assignment_loss‚4.055586081963998 2loss/SRqqt2/assignment_loss‚2.4695586719011007 /loss/FBt3/detection_loss‚0.7646950241457329 epoch‚0 /loss/FRt1/assignment_loss‚5.264029935348866 /loss/FRt1/detection_loss‚2.6683604259823643 .loss/FBt4/detection_loss‚8.117064952850342 2loss/SRqqt1/assignment_loss‚2.9412312686443327 0loss/FBt1/assignment_loss‚0.6300589523937864 1loss/FBt4/assignment_loss‚0.40394023060798645 1loss/SRqqt2/detection_loss‚3.4326040305589376 %loss/total_loss‚66.31731884224777 1loss/SRqqt4/assignment_loss‚2.210981059610174ê“þÄ\‘Ö 1loss/SRqqt4/assignment_loss‚2.210981059610174 _step‚83 /loss/FRt2/assignment_loss‚5.829993373469303 /loss/FRt4/detection_loss‚2.4200611655358917 .loss/FBt1/detection_loss‚4.926826904217402 trainer/global_step‚1049 _runtime‚ 770.730738561 2loss/SRqqt3/assignment_loss‚2.1466389216263506 1loss/FBt2/assignment_loss‚0.48692540195939915 0loss/FBt3/assignment_loss‚0.2628579608636334 /loss/FRt2/detection_loss‚2.6976256370544434 /loss/FRt3/detection_loss‚2.2740715390190167 1loss/SRqqt3/detection_loss‚1.7009711454019827 1loss/SRqqt4/detection_loss‚1.5778165957733485 /loss/FBt2/detection_loss‚3.1522061033495543 1loss/SRqqt1/detection_loss‚2.5274138674139977 $ _timestamp‚1.788405667743369e+09 0loss/FRt3/assignment_loss‚3.3557995925558375 /loss/FRt4/assignment_loss‚4.055586081963998 2loss/SRqqt2/assignment_loss‚2.4695586719011007 /loss/FBt3/detection_loss‚0.7646950241457329 epoch‚0 /loss/FRt1/assignment_loss‚5.264029935348866 /loss/FRt1/detection_loss‚2.6683604259823643 .loss/FBt4/detection_loss‚8.117064952850342 2loss/SRqqt1/assignment_loss‚2.9412312686443327 0loss/FBt1/assignment_loss‚0.6300589523937864 1loss/FBt4/assignment_loss‚0.40394023060798645 1loss/SRqqt2/detection_loss‚3.4326040305589376 %loss/total_loss‚66.31731884224777Sš„}’‚(:… ­ÏãÔ‚è gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßû:“‚(:¯ ­ÏãÔÙ¬ß $ disk.nvme0c0n1.out‚ 1402.91015625 disk.nvme0n1.in‚ 4.16796875 disk.dm-0.in‚ 4.16796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 3.1669921875 disk.dm-1.in‚0 disk.sda3.out‚34.625 disk.drbd2080.in‚0) proc.memory.percent‚16.51601791381836 cpu‚25.06893027108022 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2705.984375% disk./.usageGB‚234.84928131103516 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 4.16796875 disk.drbd1716.out‚0 disk.dm-0.out‚ 1402.91015625 network.sent‚ 1.282415e+06 disk.sda3.in‚0$ memory_percent‚98.54252338409424* disk./.usagePercent‚15.760932320455566 disk.drbd2080.out‚0 proc.cpu.threads‚17 network.recv‚352769 disk.dm-1.out‚34.625 disk.drbd2265.in‚0 disk.sda2.in‚0" disk.nvme0n1.out‚ 1402.91015625 disk.drbd2265.out‚0 disk.sda.out‚34.625 disk.drbd2266.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 238.79296875“û÷”‚(:’ ¼ÏãÔš¶· gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.376999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠÚÆ K•‚(:À ¼ÏãÔ– Œ  disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.16796875$ memory_percent‚98.54111671447754 disk.sda3.out‚ 35.140625* disk./.usagePercent‚15.760927456312418 disk.sda1.out‚0 disk.sda.out‚ 35.140625 disk.sda3.in‚0' proc.memory.availableMB‚ 239.0234375 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2705.984375$ disk./.usageGB‚234.8492088317871 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.sent‚ 1.331388e+06 disk.rbd1.out‚ 3.1669921875 disk.nvme0n1.in‚ 4.16796875 disk.dm-1.in‚0 disk.drbd2080.out‚0 cpu‚25.068561347800028 network.recv‚357665' disk.nvme0c0n1.out‚1403.29638671875 disk.drbd1716.in‚0% disk.nvme0n1.out‚1403.29638671875 disk.dm-1.out‚ 35.140625) proc.memory.percent‚16.51601791381836 disk.nvme0c0n1.in‚ 4.16796875" disk.dm-0.out‚1403.29638671875 disk.rbd0.in‚0 `€T0–+ _wandbruntime‚806  _runtime‚806îyçmÇ—Á _step‚84 ' lr-AdamW/pg1‚0.00025003299896524654 ' lr-AdamW/pg2‚0.00025003299896524654 trainer/global_step‚1099 % _timestamp‚1.7884057035707643e+09 _runtime‚ 806.558210523°Ç"Å˘Š' lr-AdamW/pg2‚0.00025003299896524654 trainer/global_step‚1099 % _timestamp‚1.7884057035707643e+09 _runtime‚ 806.558210523 _step‚84 ' lr-AdamW/pg1‚0.00025003299896524654Tp´l0™+ _wandbruntime‚806  _runtime‚806±ÿWš»'šµO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20368290 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884057035714965e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62907348001280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14078655 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60410244043776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5088153 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚1099 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 901437187072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 902092782592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15425196 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61308352382642 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15425196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20368290 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚85 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62005255218688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8990502 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68785311186944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 949032776192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67836442801664 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67836466741760 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62906674402816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62005255218688 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8990500 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62907348001280 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 901437187072 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 898779808434 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60410227004416 _runtime‚ 806.559137572 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62005237215744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 902092782592 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62906674402816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15425198 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15425198 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14078176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 ë&›¹O XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61309023852210 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35793486 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35812028 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5087676 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62906674402816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62005255218688 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8990500 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62907348001280 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 901437187072 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 898779808434 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60410227004416 _runtime‚ 806.559137572 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62005237215744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 902092782592 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62906674402816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15425198 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15425198 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14078176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20368290 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884057035714965e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62907348001280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14078655 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60410244043776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5088153 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚1099 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 901437187072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 902092782592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.s—=’™mall_pool.peak‚ 185637376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15425196 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61308352382642 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15425196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20368290 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚85 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62005255218688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8990502 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68785311186944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 949032776192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67836442801664 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67836466741760 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14091021 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15439214 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60465131134976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20405310 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8998668 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61364744417942 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚1099 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14091479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62963846869504 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8998670 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35825956 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 949895930880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62061590254592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62061590254592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 902256614912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35825956 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20386744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884057042605023e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62061608257536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62964520481280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67898117479424 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 898941813190 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20386744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5092353 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949737844736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 902912223744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20405310 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15439212 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 899596243606 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35825956 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20386744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884057042605023e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62061608257536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62964520481280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67898117479424 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 898941813190 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20386744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5092353 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949737844736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 902912223744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20405310 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15439212 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 899596243606 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14091021 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15439214 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60465131134976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704V¡Œ6`0Ÿ+ _wandbruntime‚807  _runtime‚807qþ«B ¼ /loss/FRt1/assignment_loss‚5.425659473737081 2loss/SRqqt1/assignment_loss‚2.5863547969837577 loss/FBt4/assignment_loss‚0 /loss/FRt4/detection_loss‚2.2224696607425294 1loss/SRqqt2/detection_loss‚3.0294764685630797 .loss/FBt4/detection_loss‚5.624695062637329 trainer/global_step‚1099 /loss/FRt2/assignment_loss‚5.461619035402934 /loss/FBt3/detection_loss‚0.8448290166350285 epoch‚0 /loss/FRt2/detection_loss‚2.8613920152187347 0loss/SRqqt1/detection_loss‚2.354420191171218 /loss/FBt2/detection_loss‚3.1205204824606576 /loss/FRt4/assignment_loss‚4.351197910719905 0loss/SRqqt3/detection_loss‚1.535821250786907 _step‚87 1loss/SRqqt2/assignment_loss‚2.552043566703796 1loss/FBt3/assignment_loss‚0.34593174518775444 /loss/FRt1/detection_loss‚3.2015585720539095 1loss/SRqqt4/assignment_loss‚2.046065554022789 0loss/FBt2/assignment_loss‚0.5914978803119387 _runtime‚ 807.249690044 /loss/FRt3/assignment_loss‚3.281036560352032 /loss/FRt3/detection_loss‚2.0741739765955853 0loss/SRqqt4/detection_loss‚1.534644790155732 .loss/FBt1/detection_loss‚4.006423379693713 % _timestamp‚1.7884057042623208e+09 2loss/SRqqt3/assignment_loss‚2.0986434968939998 0loss/FBt1/assignment_loss‚0.3953763785456026 %loss/total_loss‚61.54585126557601¿OúàF¡À 1loss/SRqqt2/assignment_loss‚2.552043566703796 1loss/FBt3/assignment_loss‚0.34593174518775444 /loss/FRt1/detection_loss‚3.2015585720539095 1loss/SRqqt4/assignment_loss‚2.046065554022789 0loss/FBt2/assignment_loss‚0.5914978803119387 _runtime‚ 807.249690044 /loss/FRt3/assignment_loss‚3.281036560352032 /loss/FRt3/detection_loss‚2.0741739765955853 0loss/SRqqt4/detection_loss‚1.534644790155732 .loss/FBt1/detection_loss‚4.006423379693713 % _timestamp‚1.7884057042623208e+09 2loss/SRqqt3/assignment_loss‚2.0986434968939998 0loss/FBt1/assignment_loss‚0.3953763785456026 %loss/total_loss‚61.54585126557601 /loss/FRt1/assignment_loss‚5.425659473737081 2loss/SRqqt1/assignment_loss‚2.5863547969837577 loss/FBt4/assignment_loss‚0 /loss/FRt4/detection_loss‚2.2224696607425294 1loss/SRqqt2/detection_loss‚3.0294764685630797 .loss/FBt4/detection_loss‚5.624695062637329 trainer/global_step‚1099 /loss/FRt2/assignment_loss‚5.461619035402934 /loss/FBt3/detection_loss‚0.8448290166350285 epoch‚0 /loss/FRt2/detection_loss‚2.8613920152187347 0loss/SRqqt1/detection_loss‚2.354420191171218 /loss/FBt2/detection_loss‚3.1205204824606576 /loss/FRt4/assignment_loss‚4.351197910719905 0loss/SRqqt3/detection_loss‚1.535821250786907 _step‚87WoÎcE¢‚(:… ËÏãÔ³îù gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×É3F1£‚(:¦ ËÏãÔ¶àü  disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# memory_percent‚98.5426664352417 disk.rbd1.out‚ 3.19921875 disk.dm-1.in‚0 disk.dm-1.out‚39 disk.sda.out‚39 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 network.recv‚361531 disk.dm-0.in‚ 4.16796875 disk.drbd2266.out‚0 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2705.984375 cpu‚25.068174634292824* disk./.usagePercent‚15.760923104184336$ disk./.usageGB‚234.8491439819336$ disk.nvme0n1.out‚1411.2783203125 disk.drbd2265.out‚0 disk.drbd1716.in‚0! disk.dm-0.out‚1411.2783203125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚39( proc.memory.availableMB‚ 238.76953125 network.sent‚ 1.337322e+06 disk.nvme0n1.in‚ 4.16796875 disk.drbd2266.in‚0 disk.sda1.in‚0) proc.memory.percent‚16.51601791381836 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 4.16796875& disk.nvme0c0n1.out‚1411.2783203125.žTϤ‚(:’ ÚÏãÔ”›— gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.355' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.270999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fT šO¥‚(:Ä ÚÏãÔó´û  disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚1411.5283203125 disk.drbd2080.out‚0& disk.nvme0c0n1.out‚1411.5283203125 disk.drbd1716.in‚0 disk.rbd1.out‚ 3.84765625 disk.nvme0n1.in‚ 4.16796875 disk.sda1.in‚0$ disk./.usageGB‚234.8491439819336 disk.drbd1716.out‚0 disk.rbd1.in‚0! disk.dm-0.out‚1411.5283203125 disk.dm-1.in‚0 disk.sda3.out‚ 39.41015625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.54090213775635( proc.memory.availableMB‚ 239.05859375# proc.memory.rssMB‚ 2705.98828125 cpu‚25.067620286149268 disk.nvme0c0n1.in‚ 4.16796875 disk.dm-0.in‚ 4.16796875 disk.dm-1.out‚ 39.41015625 disk.sda3.in‚0 disk.drbd2266.in‚0 network.sent‚ 1.386842e+06* disk./.usagePercent‚15.760923104184336 disk.drbd2265.in‚0 disk.sda.out‚ 39.41015625) proc.memory.percent‚16.51604175567627 proc.cpu.threads‚17 network.recv‚367234 disk.sda1.out‚0Ó[œ¦‚(:‘ éÏãÔΕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.858' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.77160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'IŸO§‚(:Ä éÏãÔ¿ŸÑ  disk.nvme0c0n1.in‚ 4.16796875' disk.nvme0c0n1.out‚1412.55517578125 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 3.92578125% disk.nvme0n1.out‚1412.55517578125) disk./.usagePercent‚15.76092464022954 disk.drbd1716.out‚0# memory_percent‚98.5429048538208 network.recv‚371997 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.16796875 network.sent‚ 1.393173e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 40.08984375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda.in‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 238.73046875# proc.memory.rssMB‚ 2705.98828125) proc.memory.percent‚16.51604175567627 disk.drbd1716.in‚0 disk.dm-0.in‚ 4.16796875 disk.sda3.out‚ 40.08984375 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.out‚ 40.08984375 disk.rbd0.in‚0 cpu‚25.06762368985857$ disk./.usageGB‚234.8491668701172 disk.rbd1.in‚0" disk.dm-0.out‚1412.55517578125 disk.drbd2265.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17B¦Â0¨+ _wandbruntime‚843  _runtime‚843ᜟúÄ©¾ $ _timestamp‚1.788405740263335e+09 _runtime‚ 843.250760309 _step‚88 & lr-AdamW/pg1‚0.0002614084766251759 & lr-AdamW/pg2‚0.0002614084766251759 trainer/global_step‚1149¾Wá›ÈªÂ & lr-AdamW/pg2‚0.0002614084766251759 trainer/global_step‚1149 $ _timestamp‚1.788405740263335e+09 _runtime‚ 843.250760309 _step‚88 & lr-AdamW/pg1‚0.0002614084766251759XÀ\¦Ô&«+ _wandbruntime‚843  _runs time‚843ì™Ð»'¬µO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71911221374976 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992217750016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 % _timestamp‚1.7884057402640107e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64822889158144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14719787 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64822907161088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63155450571776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65765985749504 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 943078588416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5320423 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64822889158144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14719323 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 938960717634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70918979684864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65765312151040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 942422992896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9398902 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16125998 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ÈŠŒ¿'­¹O G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71911221374976 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992217750016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 % _timestamp‚1.7884057402640107e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64822889158144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14719787 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64822907161088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63155450571776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65765985749504 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 943078588416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5320423 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64822889158144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14719323 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 938960717634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70918979684864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65765312151040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 942422992896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9398902 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16125998 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088Y]›¥®0®+ _wandbruntime‚843  _runtime‚843ÎÆ%ŒÅ&¯¿M G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37452808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71973550156800 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64150786285350 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9407068 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 992919734272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37452808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 943898029568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70980654362624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37471376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65823158229504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14732558 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21331362 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21312796 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71973740571648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64150114815574 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚1149 % _timestamp‚1.7884057409447892e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16140012 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 943242420736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 943898029568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407070 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21312796 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚ 843.932569282 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64879242196992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70980630422528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14732087 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993086209024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5325488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63210354702336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64879242196992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704’€È] °ÃM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9407068 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 992919734272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37452808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 943898029568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70980654362624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37471376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65823158229504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14732558 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21331362 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21312796 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71973740571648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64150114815574 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚1149 % _timestamp‚1.7884057409447892e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16140012 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 943242420736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 943898029568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407070 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21312796 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚ 843.932569282 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64879242196992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70980630422528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14732087 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993086209024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5325488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63210354702336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64879242196992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37452808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71973550156800 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64150786285350Z˜S´·0±+ _wandbruntime‚843  _runtime‚843Ïé"@²º /loss/FRt1/assignment_loss‚5.122355320874383 1loss/SRqqt2/detection_loss‚2.9930883145332334 0loss/SRqqt3/detection_loss‚1.835904110385024 %loss/total_loss‚59.21195717932883 0loss/FRt2/assignment_loss‚5.2118004639943445 0loss/FBt1/assignment_loss‚0.5105294447097953 /loss/FBt3/detection_loss‚1.1269906812617871 2loss/SRqqt4/assignment_loss‚1.9994573033117986 loss/FBt4/assignment_loss‚0 /loss/FRt1/detection_loss‚2.6847036393249737 1loss/SRqqt1/detection_loss‚2.8122346736490726 /loss/FBt1/detection_loss‚4.3854074411922035 1loss/SRqqt3/assignment_loss‚2.465532597640286 1loss/FBt3/assignment_loss‚0.22335621384848833 /loss/FRt4/detection_loss‚1.9085949574198042 epoch‚0 .loss/FRt2/detection_loss‚2.672410309314728 2loss/SRqqt2/assignment_loss‚2.1865693879127504 -loss/FBt2/detection_loss‚2.98497580669143 _step‚91 .loss/FRt3/assignment_loss‚3.11456362990772 0loss/SRqqt4/detection_loss‚1.420609139560998 .loss/FBt4/detection_loss‚4.276177644729614 $ _timestamp‚1.788405740946699e+09 /loss/FRt4/assignment_loss‚3.807229914835521 2loss/SRqqt1/assignment_loss‚2.9920812249183655 0loss/FBt2/assignment_loss‚0.3891478919566783 /loss/FRt3/detection_loss‚2.0882370673558293 _runtime‚ 843.93407863 trainer/global_step‚1149÷;ÁiD³¾ .loss/FRt3/assignment_loss‚3.11456362990772 0loss/SRqqt4/detection_loss‚1.420609139560998 .loss/FBt4/detection_loss‚4.276177644729614 $ _timestamp‚1.788405740946699e+09 /loss/FRt4/assignment_loss‚3.807229914835521 2loss/SRqqt1/assignment_loss‚2.9920812249183655 0loss/FBt2/assignment_loss‚0.3891478919566783 /loss/FRt3/detection_loss‚2.0882370673558293 _runtime‚ 843.93407863 trainer/global_step‚1149 /loss/FRt1/assignment_loss‚5.122355320874383 1loss/SRqqt2/detection_loss‚2.9930883145332334 0loss/SRqqt3/detection_loss‚1.835904110385024 %loss/total_loss‚59.21195717932883 0loss/FRt2/assignment_loss‚5.2118004639943445 0loss/FBt1/assignment_loss‚0.5105294447097953 /loss/FBt3/detection_loss‚1.1269906812617871 2loss/SRqqt4/assignment_loss‚1.9994573033117986 loss/FBt4/assignment_loss‚0 /loss/FRt1/detection_loss‚2.6847036393249737 1loss/SRqqt1/detection_loss‚2.8122346736490726 /loss/FBt1/detection_loss‚4.3854074411922035 1loss/SRqqt3/assignment_loss‚2.465532597640286 1loss/FBt3/assignment_loss‚0.22335621384848833 /loss/FRt4/detection_loss‚1.9085949574198042 epoch‚0 .loss/FRt2/detection_loss‚2.672410309314728 2loss/SRqqt2/assignment_loss‚2.1865693879127504 -loss/FBt2/detection_loss‚2.98497580669143 _step‚91[ú§O´‚(:’ øÏãÔ¢Þ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.529' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.105799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-HKMµ‚(: øÏãÔØ‹ñ  disk.dm-0.out‚1421.974609375 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 41.79296875 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 4.17578125 disk.sda.in‚0( proc.memory.availableMB‚ 238.71484375 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 4.55078125 disk.dm-0.in‚ 4.17578125 disk.drbd2265.out‚0) proc.memory.percent‚16.51604175567627 cpu‚25.067024706124936 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 41.79296875# proc.memory.rssMB‚ 2705.98828125 network.sent‚ 1.398467e+06 disk.drbd1716.in‚0$ memory_percent‚98.54300022125244 proc.cpu.threads‚17* disk./.usagePercent‚15.761005538610359% disk./.usageGB‚234.85037231445312 disk.sda3.in‚0 disk.sda3.out‚ 41.79296875 disk.sda2.out‚0# disk.nvme0n1.out‚1421.974609375 network.recv‚375960 disk.sda1.in‚0% disk.nvme0c0n1.out‚1421.974609375 disk.nvme0n1.in‚ 4.17578125‚…¯§’¶‚(:‡ ‡ÐãÔÊ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.098' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ?úM·‚(: ‡ÐãÔ”ÂÚ  disk.drbd2080.out‚0 disk.sda2.out‚0 network.sent‚ 1.448246e+06! disk.dm-0.out‚1426.0283203125 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 cpu‚25.06669146252487 disk.rbd1.out‚ 4.62890625 disk.sda3.in‚0$ memory_percent‚98.54474067687988' proc.memory.availableMB‚ 238.4296875# proc.memory.rssMB‚ 2705.99609375 proc.cpu.threads‚17 network.recv‚381365 disk.sda2.in‚0 disk.sda3.out‚ 43.41796875$ disk./.usageGB‚234.8503761291504 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 43.41796875* disk./.usagePercent‚15.761005794617894 disk.nvme0c0n1.in‚ 4.17578125 disk.rbd1.in‚0$ disk.nvme0n1.out‚1426.0283203125 disk.sda.out‚ 43.41796875 disk.drbd2266.out‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚1426.0283203125 disk.nvme0n1.in‚ 4.17578125 disk.sda.in‚0 disk.drbd2266.in‚0) proc.memory.percent‚16.51608943939209 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 4.175781254ã£:0¸+ _wandbruntime‚879  _runtime‚879ð›jŹ¿ & lr-AdamW/pg1‚0.0002727839542851052 & lr-AdamW/pg2‚0.0002727839542851052 trainer/global_step‚1199 % _timestamp‚1.7884057765203736e+09 _runtime‚ 879.507915377 _step‚92x<‡:ɺà % _timestamp‚1.7884057765203736e+09 _runtime‚ 879.507915377 _step‚92 & lr-AdamW/pg1‚0.0002727839542851052 & lr-AdamW/pg2‚0.0002727839542851052 trainer/global_step‚1199\›þ(0»+ _wandbruntime‚879  _runtime‚879 §-¼'¼¶O PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16826796 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5552019 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39047190 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚67640559103488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1035145711616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16826796 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22220394 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22238934 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983408798720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66881107587026 trainer/global_step‚1199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5552497 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 979796057042 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68623949899264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9807302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16826798 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75036662279680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 879.50877571 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9807300 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884057765211377e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74001516568064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1035310102528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973Ñás "½ºO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16826796 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5552019 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39047190 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚67640559103488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1035145711616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16826796 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22220394 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22238934 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983408798720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66881107587026 trainer/global_step‚1199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5552497 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 979796057042 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68623949899264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9807302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16826798 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75036662279680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_trai·'öµn_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 879.50877571 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9807300 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884057765211377e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74001516568064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1035310102528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973]¼—\Ê0¾+ _wandbruntime‚880  _runtime‚880+È.Æ&¿ÀM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68681795977728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68681122365952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66936156682982 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9815470 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 984228226560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66936828152758 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036010489344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22238848 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22238848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15372096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74063167305728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚94 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39098228 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 984883835392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚1199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68681795977728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22257414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 981266922422 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67696912142336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65955544190976 _runtime‚ 880.19459571 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22257414 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 984883835392 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67696912142336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16840814 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16840812 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984228226560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430Áùú#Ê&ÀÄM TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 984228226560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66936828152758 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036010489344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22238848 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22238848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15372096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74063167305728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚94 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39098228 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 984883835392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚1199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68681795977728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22257414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 981266922422 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67696912142336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65955544190976 _runtime‚ 880.19459571 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22257414 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 984883835392 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67696912142336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16840814 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16840812 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984228226560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68681795977728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68681122365952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66936156682982 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9815470 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477^©0Á+ _wandbruntime‚880  _runtime‚880¬AbV=· _runtime‚ 880.195956998 .loss/FRt3/assignment_loss‚2.94986368786721 1loss/SRqqt4/detection_loss‚1.5016766069339047 /loss/FRt2/assignment_loss‚5.036916820208232 loss/FBt4/assignment_loss‚0 .loss/FBt4/detection_loss‚5.247579574584961 &loss/total_loss‚62.114982902590405 1loss/FBt2/assignment_loss‚0.47089328791995894 _step‚95 $ _timestamp‚1.788405777208373e+09 2loss/SRqqt1/assignment_loss‚2.7047809428638883 0loss/FBt1/assignment_loss‚0.6835080257483891 /loss/FRt3/detection_loss‚2.4872808498995647 .loss/SRqqt1/detection_loss‚2.2803036292394 epoch‚0 1loss/SRqqt3/assignment_loss‚2.058698375357522 /loss/FRt2/detection_loss‚2.9813955585161844 .loss/FRt4/detection_loss‚2.421507814416179 /loss/FBt3/detection_loss‚0.8689228210718764 /loss/FRt4/assignment_loss‚4.715818288149657 0loss/SRqqt2/detection_loss‚2.365620857477188 trainer/global_step‚1199 0loss/SRqqt2/assignment_loss‚2.92730218693614 2loss/SRqqt4/assignment_loss‚2.4237736073098604 .loss/FBt1/detection_loss‚4.799327969551086 /loss/FRt1/assignment_loss‚4.579951524734497 0loss/FBt3/assignment_loss‚0.3548102379922802 /loss/FRt1/detection_loss‚2.7595265539068925 1loss/SRqqt3/detection_loss‚1.8711349036958482 .loss/FBt2/detection_loss‚3.624388778209686²ŽŒµAû /loss/FRt3/detection_loss‚2.4872808498995647 .loss/SRqqt1/detection_loss‚2.2803036292394 epoch‚0 1loss/SRqqt3/assignment_loss‚2.058698375357522 /loss/FRt2/detection_loss‚2.9813955585161844 .loss/FRt4/detection_loss‚2.421507814416179 /loss/FBt3/detection_loss‚0.8689228210718764 /loss/FRt4/assignment_loss‚4.715818288149657 0loss/SRqqt2/detection_loss‚2.365620857477188 trainer/global_step‚1199 0loss/SRqqt2/assignment_loss‚2.92730218693614 2loss/SRqqt4/assignment_loss‚2.4237736073098604 .loss/FBt1/detection_loss‚4.799327969551086 /loss/FRt1/assignment_loss‚4.579951524734497 0loss/FBt3/assignment_loss‚0.3548102379922802 /loss/FRt1/detection_loss‚2.7595265539068925 1loss/SRqqt3/detection_loss‚1.8711349036958482 .loss/FBt2/detection_loss‚3.624388778209686 _runtime‚ 880.195956998 .loss/FRt3/assignment_loss‚2.94986368786721 1loss/SRqqt4/detection_loss‚1.5016766069339047 /loss/FRt2/assignment_loss‚5.036916820208232 loss/FBt4/assignment_loss‚0 .loss/FBt4/detection_loss‚5.247579574584961 &loss/total_loss‚62.114982902590405 1loss/FBt2/assignment_loss‚0.47089328791995894 _step‚95 $ _timestamp‚1.788405777208373e+09 2loss/SRqqt1/assignment_loss‚2.7047809428638883 0loss/FBt1/assignment_loss‚0.6835080257483891_0{|{’Ä‚(:‡ –Ðãԕͪ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\óðàKÅ‚(:À –ÐãÔ몆  disk.drbd2265.out‚0 network.recv‚385896 disk.dm-0.in‚ 4.17578125 disk.dm-1.in‚0 disk.drbd2266.out‚0& disk.nvme0c0n1.out‚1426.3134765625 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚ 4.17578125) proc.memory.percent‚16.51608943939209 network.sent‚ 1.454043e+06% disk./.usageGB‚234.85084533691406 disk.nvme0n1.in‚ 4.17578125 disk.sda.in‚0 disk.sda3.out‚ 43.92578125 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0$ memory_percent‚98.54469299316406 disk.rbd1.out‚ 4.73828125$ disk.nvme0n1.out‚1426.3134765625 disk.sda3.in‚0 disk.sda2.in‚0$ proc.memory.availableMB‚238.4375# proc.memory.rssMB‚ 2705.99609375 cpu‚25.06645363402177! disk.dm-0.out‚1426.3134765625 disk.sda.out‚ 43.92578125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 43.92578125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0* disk./.usagePercent‚15.761037283544605 disk.rbd0.out‚0 disk.sda1.out‚0èM3Æ‚(:’ ¥ÐãÔÒû‰ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.444' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.888800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜¥ßLÇ‚(:Á ¥ÐãÔÝ˽ % disk.nvme0c0n1.out‚1426.669921875 disk.dm-0.in‚ 4.17578125( proc.memory.availableMB‚ 145.81640625 network.sent‚ 1.503931e+06 disk.drbd2080.out‚0 network.recv‚391357* disk./.usagePercent‚15.761018338987071 disk.rbd1.in‚0 disk.dm-1.out‚ 44.66015625 disk.sda3.out‚ 44.66015625) proc.memory.percent‚17.07942485809326$ disk./.usageGB‚234.8505630493164 disk.nvme0c0n1.in‚ 4.17578125 disk.nvme0n1.in‚ 4.17578125 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚1426.669921875 disk.dm-0.out‚1426.669921875 disk.sda.out‚ 44.66015625 disk.sda1.out‚0$ memory_percent‚99.11000728607178 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 4.73828125 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2798.29296875 cpu‚25.137027627576686 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0£Š;¬œÈ‚(:‘ ´ÐãÔ±“¥ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.341' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.868199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹½Šù@É‚(:µ ´ÐãÔÚŽ’  disk.rbd1.in‚0 disk.dm-0.in‚4.203125 disk.drbd2265.in‚0) proc.memory.percent‚17.14019775390625 proc.cpu.threads‚17$ disk.nvme0n1.out‚1429.9052734375 disk.sda.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 proc.memory.rssMB‚2808.25$ disk./.usageGB‚236.8614387512207 disk.sda1.out‚0% proc.memory.availableMB‚ 135.71875 disk.rbd0.in‚0 disk.sda2.out‚0 network.recv‚395160 disk.nvme0n1.in‚4.203125 disk.sda.out‚ 45.14453125 disk.sda3.in‚0 disk.drbd2080.out‚0$ memory_percent‚99.17163848876953 network.sent‚ 1.509163e+06 disk.dm-1.in‚0 disk.sda3.out‚ 45.14453125 disk.drbd2266.out‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚1429.9052734375 disk.rbd1.out‚ 4.80078125 disk.rbd0.out‚0* disk./.usagePercent‚15.895969894578965 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.203125! disk.dm-0.out‚1429.9052734375 disk.dm-1.out‚ 45.14453125 disk.drbd2265.out‚0 disk.drbd2266.in‚0 cpu‚25.2145886846359773'ŸÊ‚(:’ ÃÐãÔñÆ gpu.0.gpu‚53.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.840600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù;AOË‚(:Ä ÃÐãÔ“ª÷  disk.sda.out‚ 45.73046875 disk.sda3.out‚ 45.73046875 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚99.17168617248535* proc.memory.percent‚17.140555381774902 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.80078125 disk.nvme0n1.in‚ 7.2890625 disk.drbd2266.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚2789.08447265625 network.recv‚398395 disk.drbd2080.out‚0 disk.sda1.out‚0% disk.nvme0n1.out‚2789.08447265625" disk.dm-0.out‚2789.08447265625 disk.sda.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 7.2890625) disk./.usagePercent‚15.85020419570543% disk./.usageGB‚236.17949676513672 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 45.73046875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 135.7109375# proc.memory.rssMB‚ 2808.30859375 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 cpu‚25.293426154502846 network.sent‚ 1.514275e+06 disk.dm-0.in‚ 7.2890625HYÛ+cÌb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JP(ánÚsÍbn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JPÉlÎbg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JPýö oÏbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPp`θgÐbb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JP¢‘¡ŸqÑbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP:>tªtÒbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPT¢ÊzÓbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP}‡átÔbo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP8qn;yÕbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPDñ‹ßhÖbc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JPæl×bg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JP¢žàórØbm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JPŸ Fû|Ùbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPo""zÚbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPía.åmÛbh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPëEÄ\yÜbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÃâqsÝbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP„K­áxÞbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPàJjfßba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP›'üoàbj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JPÙtsÛábi OšÈhFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP¯omâbh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JPõ;ù/sãbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPE¥7qäbl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JP)¬óþvåbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPäÀX—kæbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JP€’vçbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPö€Òièbd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPnÿ®Ømébh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JP–)Øwkêbf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JPAŽnÆoëbj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JPÇml+oìbj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JPúûÆ©víbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP¤tîbo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP£7/€sïbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP9%â/dðb_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JP ¨QAl™bg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JP‡Ä‡mšbh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JP—'§Nv›bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP9Z/•kœbf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPÇÔybt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPN(Urmžbh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPð=¯…wŸbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP¾¿ý[q bl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPÖ¦öep¡bk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JPB±\w¢br ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP½VƒØd£b_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚22766060 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚16039249 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚68179092620288 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492427776 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚16039405 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490911116 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71018415323648 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚40277703 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚71018415323648 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1010164851200 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚22630 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚71018925754368 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚154 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚17496452 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚77535845274112 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71018415323648 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚40277703 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚71018415323648 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1010164851200 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚22630 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚71018925754368 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚154 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚17496452 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚77535845274112 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚22766060 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚16039249 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚68179092620288 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492427776 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚16039405 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490911116 FÁGmÒbh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JP†\k!nÓbi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JP¶oŽkÔbf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPè 5itÕbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP¿ü'œpÖbk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPñØ>f×ba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP¼¢ºdØb_ DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPêá§qc‡b^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JP¶í¡iˆbd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JP|ì,n‰bi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JP)óegŠbb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPÏÝÑìp‹bk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JP°I™nŒbi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPÃmÊtbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP÷%VÄgŽbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JPäD’†lbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JPã)(Hnbi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JPŒŸ—z‘bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP—™¤i’bd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JPå(ïþr“bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP}†±j”be BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP,Õ„ßo•bj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JPÓBÚq–bl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPº–øªv—bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP\ £k˜bf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JP”šqÉr™bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPù©#×qšbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPæ* Pt›bo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPUkC~hœbc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JP' £jbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JP»< ‘ožbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPøS“pŸbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPz›ÅŸs bn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP6ã?t¡bo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP&:À¦c¢b^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JPn†:Üu£bp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP1ù|[¤bV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JP}Юþn¥bi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JP ðõu¦bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP¹¡ç²a§b\ 9DeviceStatsMonitor.on_validation_batch_end/max_split_size"trainer/global_step2:JPµÎ°Ös¨bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPÓg |©bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP*Ußlªbg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JP~{ÜLe«b` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPz±-5j¬be BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JPs@ãi­bd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JP•G up®bk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JP¦PŒv¯bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP…)•Vo°bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPÙB‡°w±br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚22788553 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚40280618 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚17507256 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚22773362 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111552000 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 _runtime‚ 935.27045158 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚10214180 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1010037550080 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1010037550080 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚77574221626368 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17507258 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492427776 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚68214369340416 _step‚97 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1062387066368 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚69220689751226 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15193 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚22773362 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚71055610479104 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15191 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚5836534 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863048704 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15193 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚5836379 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1010529977856 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚17507258 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚16050716 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9954819072 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚70045572929024 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚70045572929024 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643691477 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚40280618 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚16050559 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚76511810619904 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490911116 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚68214386379776 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚40295811 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71055610479104 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚22630 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510430720 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510430720 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1010529977856 [HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚71056120909824 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚22788553 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚40280618 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚17507256 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚22773362 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111552000 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 _runtime‚ 935.27045158 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚10214180 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1010037550080 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1010037550080 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚77574221626368 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17507258 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492427776 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚68214369340416 _step‚97aü1Ü‘È‚(:† ÒÐãÔˆÕâ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚185.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü÷ý‰MÉ‚(: ÒÐãÔÝÎá  disk.dm-0.in‚8.34375 disk.drbd2266.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚2790.45947265625" disk.dm-0.out‚2790.45947265625 disk.dm-1.out‚ 46.19921875 disk.sda3.out‚ 46.19921875 proc.cpu.threads‚17 disk.dm-1.in‚0# proc.memory.rssMB‚ 2808.41015625 cpu‚25.360179024515908 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.141175270080566 disk.rbd1.out‚ 4.9423828125 disk.sda.out‚ 46.19921875$ memory_percent‚99.16942119598389 network.recv‚402972 disk.rbd0.out‚0 disk.sda1.out‚0* disk./.usagePercent‚15.850215972052005% disk./.usageGB‚236.17967224121094 disk.nvme0c0n1.in‚8.34375' disk.nvme0c0n1.out‚2790.45947265625 disk.nvme0n1.in‚8.34375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 136.08203125 network.sent‚ 1.520162e+06 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0µ¢4SHÊbC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPözï¬NËbI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPjã^JÌbE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP_K†JÍbE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPÄ-NÎbI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP!f.7NÏbI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP–VeQJÐbE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP #zNÑbI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPßgÂHÒbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP@‹÷ØJÓbE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÝÜ—3HÔbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP/dC'JÕbE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP)¡«”HÖbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP”M~@×b; particle/accuracy_3_of_5"trainer/global_step2:JP“ô(HØbC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP¹4{uNÙbI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP‡‘ú'HÚbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÛFþƒNÛbI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP¤¡"fJÜbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPs3eÓHÝbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP²w§ÐJÞbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP…7à®JßbE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP(9—JàbE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPqƒ«JábE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPE²"]JâbE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP»ÔTNãbI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPă‚NäbI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP2þsòHåbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP™¶â«NæbI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPšn2ÒJçbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPLÈf HèbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÁÐaHébC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPdb¯»HêbC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPrTßJëbE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPq5ùENìbI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP.£¸JíbE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPòèÁJîbE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPèöt;ïb6 jet/accuracy_2_of_6"trainer/global_step2:JPû­õJðbE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPC9ÙW@ñb; particle/accuracy_2_of_7"trainer/global_step2:JP NòbI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPm§ ZHóbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPi”h_NôbI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPO>ä;õb6 jet/accuracy_2_of_4"trainer/global_step2:JPͶó @öb; particle/accuracy_4_of_5"trainer/global_step2:JP½œÛ(J÷bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÏ4lNøbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPõ¾É8HùbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPéÌEHúbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPq×+LNûbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP’|”áNübI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP¢0b@ýb; particle/accuracy_3_of_4"trainer/global_step2:JPÉG²HþbC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPÂË'HÿbC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP]øcN€bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPçæÞˆHbC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP)U³‚;‚b6 jet/accuracy_4_of_5"trainer/global_step2:JP½ÛßJƒbE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP°Væ¨H„bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPXsJ…bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP˜ÑfH†bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPíœëN‡bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPw#§ÝJˆbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPêÒoN‰bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP>²5JŠbE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÓŸBJ‹bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPYÁôYHŒbC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP>L¬JbE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP}þ€HŽbC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPyÝ—ñHbC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPÞÇTÚHbC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPû¯MJ‘bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPŽ¤k`J’bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP *¯RN“bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPk_9N”bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP_÷ H•bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP)1N–bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP‚ùX@—b; particle/accuracy_1_of_1"trainer/global_step2:JPuj òH™bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP$ýÁHšbC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPÔ ÏN›bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPqñL¨NœbI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPˆb¥JbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPáê ³NžbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¸¶Ù‚NŸbI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPçØH bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPL㤵J¡bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÙJ¢bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPØõ<1J£bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPanJ¤bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP<øuŽN¥bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPR¹H¦bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPT;@J§bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÏ°T§H¨bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP¨3¶/;©b6 jet/accuracy_2_of_3"trainer/global_step2:JPÆÃ@ªb; particle/accuracy_7_of_7"trainer/global_step2:JPôèæÔJ«bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÀ­ÛJ¬bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPlj0N­bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP9…|J®bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP%ÙkãN¯bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP3†DäN°bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÕ@mÍH±bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP—y‰ J²bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÙ¸í|J³bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÓÒW”J´bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPˆ·NµbI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¨©ÆòN¶bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPⱬýN·bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP°1“J¸bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPûÏÛH¹bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP}GgHºbC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPÒë3ƒJ»bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPø#¹;H¼bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPo.=H½bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPø ÷9J¾bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP;LJ¿bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP%2[äJÀbE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP–"ÛHÁbC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPé[JÂbE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP++ôfHÃbC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPìîžÚJÄbE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP†õ‘NÅbI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPB3ëúHÆbC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÌlÉJÇbE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPͼ(ç@Èb; particle/accuracy_4_of_4"trainer/global_step2:JPýÃOaJÉbE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPâê[HJÊbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP|ßÇHËbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPrܧyNÌbI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPлNÍbI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÂ/A¤JÎbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÆç$"JÏbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPµJ½ÎJÐbE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPèÉ JÑbE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPJÙºJÒbE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPϨozHÓbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP“ÛÍHÔbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPD‡QqHÕbC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP¡ú:HÖbC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP^˜žÏJ×bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP¾âCJØbE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP9¨æKHÙbC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP±‹¨RJÚbE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP´ý ;Ûb6 jet/accuracy_1_of_4"trainer/global_step2:JPèRÞHÜbC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP XÝJÝbE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP­A´óNÞbI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPøËÖHßbC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP’í^DJàbE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP›©–HábC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP¨5‡ÅJâbE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPW¼ÜyJãbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPYD¿BNäbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP>1L~NåbI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP0ooóHæbC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP€ÌHçbC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPn ¤NèbI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP¯æVJébE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPW\F²JêbE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPecØÐJëbE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPË°´2HìbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP2=,JíbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP?͈•HîbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÛbNHïbC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPµæaJðbE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPMø´ÂHñbC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPƒï„NòbI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPC‹NóbI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÄbvS¦èNƒ bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP>%ŒN„ bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÃVOH… bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPìIóƒJ† bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPú×H‡ bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPm‹Jˆ bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPh#lêN‰ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP6ÆJŠ bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP`úÚwN‹ bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPØî³øNŒ bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP«[»J bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP¡ð)ÉJŽ bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP÷H™J bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP½LCJ bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÚÓ;‘ b6 jet/accuracy_1_of_1"trainer/global_step2:JP²k@J’ bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP…ÂlNJ“ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP_â“ÐH” bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPBÒ;¥N• bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPœÿ¼9H– bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPîsN— bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPöé’J˜ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPDÅqó@™ b; particle/accuracy_6_of_6"trainer/global_step2:JPk…ÿNš bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP4B¸‚H› bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPùË'Jœ bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPë¶X/J bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPuFœNž bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP¸ã·ÛNŸ bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPeÚÀ!N  bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP«¹LJ¡ bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPéè4jH¢ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP ¥’öJ£ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPσN¤ bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPŒ™–<¥ b7 particle/specificity"trainer/global_step2:JP?gæN¦ bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÁc›ûH§ bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPðÊN¨ bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP)SssJ© bE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP4Jª bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPý¡y—J« bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‚£hOH¬ bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPùJ­ bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP(dÑ@® b; particle/accuracy_2_of_3"trainer/global_step2:JPt™.ÉJ¯ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPtá‘;° b6 jet/accuracy_5_of_7"trainer/global_step2:JP÷ÃÎJ± bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP, ¼TN² bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP}Å[@³ b; particle/accuracy_5_of_7"trainer/global_step2:JPZ4MaN´ bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP5âɇHµ bC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPiÂ3J¶ bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPɉ%H· bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP Fö§N¸ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPC¾Z?J¹ bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPš)Ä_Hº bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP¤ø¾yN» bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPPÜí!;¼ b6 jet/accuracy_2_of_5"trainer/global_step2:JP¢†ädH½ bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPó܇0J¾ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÏLɼJ¿ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP=*öåHÀ bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP'²JÁ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÝâ:H bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPr×/œJà bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPjÝ=“NÄ bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPí0îANÅ bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÇé‘1NÆ bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPta«JÇ bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPæd<ïHÈ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP &¬JÉ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPº+ HÊ bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP‰Æh HË bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPЈ.fHÌ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPº%wýJÍ bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPi6"ÔNÎ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP œ?—JÏ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP6V!;Ð b6 jet/accuracy_3_of_7"trainer/global_step2:JP*¯@±JÑ bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPœ,ÁÈNÒ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP“ ÒNÓ bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP‹KéHÔ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP€Ç©;Õ b6 jet/accuracy_4_of_6"trainer/global_step2:JPÖ_é¡JÖ bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP>NÆH× bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP÷-Ì–HØ bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP}¶xÝJÙ bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¤tNÚ bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPßHÛ bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP”s|HÜ bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP.  HÝ bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPs FWNÞ bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPá)åüJß bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPñ¯6üHà bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPï¢yƒNá bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPAñ~Nâ bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPóð„®Hã bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP•-cJä bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÅâSRHå bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPƒº¦ÁHæ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP9²Hç bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPˆg qJè bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP6{OŽNé bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP—v×+@ê b; particle/accuracy_3_of_3"trainer/global_step2:JPpÝv•Jë bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP8(úHì bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPMdK¤;í b6 jet/accuracy_1_of_6"trainer/global_step2:JPvçp#Nî bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPyeÈpNï bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPHÄ´;ð b6 jet/accuracy_4_of_4"trainer/global_step2:JPª/Ç­Jñ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPê¾r@ò b; particle/accuracy_5_of_6"trainer/global_step2:JP!*Hó bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP›i¡RHô bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÆöù!Nõ bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP'*0ONö bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¾ÇsüH÷ bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP P£Jø bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPake1Nù bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPø píNú bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPøŸ úNû bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP é=Hü bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÆ« ¼Hý bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP_ qÃJþ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPû]'Nÿ bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÂÓG¯N€ bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÅ…óXN bI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPF¨}°N‚ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPõ%·Hƒ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPídcN„ bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPm*+KN… bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPžw¨ÃH† bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP3%‡N‡ bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP4Þý¨Jˆ bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP¨‚ù£H‰ bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP{,ûûNŠ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÌ­‡N‹ bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP÷2]ÒHŒ bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP‡<ÖÙH bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPüc|HŽ bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPýˆŸJ bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÄñÊH bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPŠÀJ‘ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP°ãk‚J’ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPcBéRH“ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPHƒgJ” bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPüt™VN• bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP)î³N– bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPù§¶šH— bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPºdÅÎJ˜ bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPÏSˆ£N™ bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPzu$9Jš bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPL㟠N› bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPë‹C/8œ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/gëþâqlobal_step2:JPy‡úH bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPËXmJž bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPâ.ß;Ÿ b6 jet/accuracy_7_of_7"trainer/global_step2:JP,‚òŒN  bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÏÜrŒH¡ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPm¹í.N¢ bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP0®ÐþN£ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP»H¤ bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPSæìGN¥ bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPÌ3ú;¦ b6 jet/accuracy_1_of_5"trainer/global_step2:JPõ2{H§ bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPôZeYH¨ bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPrO¹N© bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP{Â6Nª bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP±ðEyH« bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¶ÔÃH¬ bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÔ<ÀêJ­ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP€NjüN® bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÆôñ2J¯ bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÐÆîJ° bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÉIá&N± bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPágRJ² bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP‰;³ b6 jet/accuracy_3_of_3"trainer/global_step2:JP×0};´ b6 jet/accuracy_3_of_5"trainer/global_step2:JPiUûJµ bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPô2ȬN¶ bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP—¨>ÄN· bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPçãëIH¸ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP˜¯ù±J¹ bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÿf£âJº bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP'mZ‹J» bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPØÌÃFJ¼ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPø%Aà@½ b; particle/accuracy_4_of_6"trainer/global_step2:JP)E"òJ¾ bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÂÓ„KN¿ bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPïßNÀ bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPZ¸;HÁ bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPi$ðJ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÀŠ]&Nà bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP—Ô?ŽNÄ bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP0]^JÅ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPEÎäJÆ bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP·,ÀJÇ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPƦ„HNÈ bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPý!ƒNÉ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP,jÙ…@Ê b; particle/accuracy_1_of_2"trainer/global_step2:JP¼pHË bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPüõ>×HÌ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPZ¸JÍ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP1£“iHÎ bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÖ®ïNÏ bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP¤Ç2üJÐ bE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP)=HÑ bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPtâ˜\JÒ bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP)ï©—NÓ bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¶IÎNÔ bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÕÇ©pNÕ bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP‰ht¤NÖ bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPƒvÅmH× bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPâ±NGHØ bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPXÉZHÙ bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP>®O8Ú b3 particle/f_score"trainer/global_step2:JPùó$¶NÛ bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPäå¿‹NÜ bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPïËpNÝ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP%}NÞ bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPl/ºJß bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP¸ð@Jà bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¥åáòJá bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPóXb9Jâ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPJ¿ bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÞ;$HÀ bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPûFVQJÁ bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŸõæoN bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP.èä9à b4 particle/accuracy"trainer/global_step2:JP'ë JÄ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPóA×JÅ bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPY°ÌyJÆ bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÙ_I&JÇ bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPýEîNÈ bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPüÓ@JÉ bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPüQ”NÊ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPS`NË bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPWs3bNÌ bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPl{½æHÍ bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPM¹¹HÎ bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP™ã›HÏ bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP6°NÐ bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPì“v&;Ñ b6 jet/accuracy_5_of_6"trainer/global_step2:JP­bõ‹JÒ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP£òŸ³JÓ bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÀ`´HÔ bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP¡ûoNÕ bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPbO]HÖ bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÔñÅáH× bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPeÄfpJØ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP Úôí@Ù b; particle/accuracy_1_of_3"trainer/global_step2:JPI¯HÚ bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP¿;Û b6 loss/val_total_loss"trainer/global_step2:JP‡ÞY•NÜ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPG”°ÛNÝ bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP”—\’NÞ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPOßShHß bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPr9@ÒJà bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÖSþgHá bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP2e>Nâ bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPá_æ3Jã bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP–Óñ\Jä bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP_a÷ËJå bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP°kxZNæ bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP]7ùHç bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPQk‘‹Hè bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP­¿núNé bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPqfÞÑNê bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPtÈÿ—Në bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPö¯cHì bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP¾®Ì½Hí bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP~(Jî bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPWö{< b7 particle/sensitivity"trainer/global_step2:JP=‡+ÐJ‘ bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP3núæJ’ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPð]”;“ b6 jet/accuracy_4_of_7"trainer/global_step2:JPXeZ N” bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP.¬ƒ¾N• bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPè1JöN– bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP?œÊJ— bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP„Г J˜ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPñŠ„J™ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPn¹4¶Nš bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP>u2ûJ› bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP??\fHœ bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP±ÆÝN bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPåsÉNž bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP2tžb;Ÿ b6 jet/accuracy_1_of_3"trainer/global_step2:JPÐ5«H  bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPò·}ìN¡ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP©Ûú$N¢ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP±ØmáN£ bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPîÃýþ@¤ b; particle/accuracy_1_of_6"trainer/global_step2:JPØU>wJ¥ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP Ó^H¦ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¨·ÙJ§ bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP§î;¨ b6 jet/accuracy_5_of_5"trainer/global_step2:JP-YåŽH© bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPDÖªzHª bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP˜ç'J« bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPàH+H¬ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPAý'N­ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÇL;š;® b6 jet/accuracy_3_of_6"trainer/global_step2:JP:=N¯ bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP—a™ J° bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÃ7mÈN± bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÊìÔN² bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPã¸ËN³ bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPß,Ñ@´ b; particle/accuracy_1_of_7"trainer/global_step2:JPQQ¿@µ b; particle/accuracy_3_of_7"trainer/global_step2:JP]s³éN¶ bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPvNJ· bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPõ :N¸ bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP=PbJ¹ bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPëHº bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPïF­xJ» bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPÂRÀZN¼ bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP÷Á™N½ bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP•fÊ%H¾ bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP†û ú@¿ b; particle/accuracy_6_of_7"trainer/global_step2:JP=šbïNÀ bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP 0³º;Á b6 jet/accuracy_2_of_2"trainer/global_step2:JP}'\ëJ bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPö9ÀgHà bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPª\ÒNÄ bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPŒƒwNÅ bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP35]HÆ bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP¤ØÆNÇ bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP€—”JÈ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPa7‚ENÉ bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPbƒ‰ßNÊ bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÔóŸ¸JË bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPr]ÕJÌ bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÄñ–½NÍ bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPߣ âJÎ bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPû¥°«HÏ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPõ˜·4HÐ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÕ0ÊNÑ bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPû“HÒ bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPºàAHÓ bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP z}wJÔ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÌ”}ÏNÕ bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP%Çß|JÖ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPóâ$àN× bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP:‡iHØ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP€ÿ}LHÙ bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPy­ZÔNÚ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP7AeNÛ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPMÕ÷íHÜ bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPgV—JÝ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPµ®JÞ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPYlJß bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPS 7Jà bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPX“u§Ná bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÖYOíNâ bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPsÇñRJã bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPŒ£þNä bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP\štîNå bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPßRœ´Jæ bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPQ+ÊHç bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP|/F]Nè bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPë –Hé bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP{ЧJê bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP&îDJë bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPW,2ÙHì bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP* ¹€Ní bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPî®CÓHî bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP?Z[ÉNï bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPΖ'åJð bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPJUŒ@ñ b; particle/accuracy_3_of_6"trainer/global_step2:JPSÙJò bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP%O5Hó bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP,#8Nô bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP$%ˆJõ bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP£ú@ö b; particle/accuracy_1_of_5"trainer/global_step2:JPŸè‚ H÷ bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPð"@ø b; particle/accuracy_4_of_7"trainer/global_step2:JP dŒ.Jù bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÙ–ìNú bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÖÃN¹Jû bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP´y—õ;ü b6 jet/accuracy_6_of_7"trainer/global_step2:JP&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3128848671913147 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 particle/accuracy_1_of_7‚1 particle/accuracy_3_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.005807508248835802 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 particle/accuracy_6_of_7‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 +jet/accuracy_2_of_2‚0.06767892837524414 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07230252027511597 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1751020848751068 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009876543655991554 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 _step‚98 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.1739978790283203 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9648424983024597 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9794096946716309 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.15419742465019226 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05525325983762741 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.18917907774448395 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08067916333675385 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.27609676122665405 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.25438594818115234 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07094291597604752 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.15330331027507782 particle/accuracy_3_of_6‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.29999998211860657 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.19711525738239288 particle/accuracy_1_of_5‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 particle/accuracy_4_of_7‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.285849928855896 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2576594948768616 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3184027373790741 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13374485075473785 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.0943983793258667 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1666063368320465 ,jet/accuracy_3_of_4‚0.026402678340673447 particle/accuracy_2_of_2‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17555353045463562 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8155224323272705 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.23598404228687286 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.23863635957241058 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_2_of_7‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.33024829626083374 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.115079365670681 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16974475979804993 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.16106584668159485 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9973545670509338 particle/accuracy_3_of_5‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9792776107788086 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.001799962599761784 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995074450969696 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1449326127767563 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.05000000074505806 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.2199999988079071 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.14413894712924957 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18097653985023499 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1481473445892334 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.983563244342804 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.16341683268547058 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0011337868636474013 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.23076923191547394 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1814814805984497 particle/accuracy_2_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.956250011920929 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.22376833856105804 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.2646174132823944 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17882905900478363 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11724557727575302 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_3_of_4‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1748754382133484 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9793145656585693 jet/accuracy_4_of_5‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.23333333432674408 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1458333283662796 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16638554632663727 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚1208 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_1‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2994663715362549 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.2857142984867096 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1458333283662796 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974084496498108 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.15603460371494293 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.17298851907253265 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0028656101785600185 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.23895929753780365 +jet/accuracy_2_of_3‚0.14665542542934418 particle/accuracy_7_of_7‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32604196667671204 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.014367816969752312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.059696830809116364 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.18774832785129547 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.09606862813234329 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1527777761220932 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07094291597604752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.17777778208255768 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13409961760044098 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.15229886770248413 particle/accuracy_4_of_4‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.31712964177131653 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.34266993403434753 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.17002059519290924 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.4545454680919647 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3296545743942261 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13733887672424316 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.44999998807907104 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.7616811394691467 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.007577079813927412 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.18891634047031403 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2705555260181427 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.1600281149148941 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.23534482717514038 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9953703880310059 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9852305054664612 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.17803767323493958 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9954648613929749 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17335672676563263 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.16049383580684662 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.17418727278709412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2571648955345154 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.19033953547477722 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.3125 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3513636887073517 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1547335535287857 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6113747358322144 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03061659261584282 )jet/accuracy_1_of_1‚0.308608740568161 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2425837367773056 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.19190643727779388 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2589065134525299 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.38017043471336365 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.13271604478359222 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.17520484328269958 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9402198195457458 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12932829558849335 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2757936716079712 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18344224989414215 & Purity/2FRt0SRqqt3FBt/FRt_purity‚1 % _timestamp‚1.7884058446391487e+09 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 particle/accuracy_5_of_7‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19051723182201385 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02380952425301075 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_2_of_5‚0.29191628098487854 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.13700349628925323 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.1590360552072525 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003968254197388887 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.2857142984867096 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2757367193698883 jet/accuracy_3_of_7‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0022694868966937065 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.21827943623065948 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.21733863651752472 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9875777959823608 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1453070342540741 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1264546811580658 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.10330472886562347 *jet/accuracy_1_of_6‚0.8269230723381042 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_4_of_4‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_6‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9889121651649475 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.1805555522441864 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 epoch‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9843008518218994 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.24321705102920532 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16653752326965332 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1666666567325592 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9885929226875305 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.19361571967601776 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.009408602491021156 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1619369387626648 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.3947368562221527 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.053030308336019516 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2321428507566452 jet/accuracy_7_of_7‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9314420819282532 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1636904776096344 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_5‚0.8062660694122314 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.04888888821005821 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.22070260345935822 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.992909848690033 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15543650090694427 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.15603460371494293 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.01259425189346075 ,jet/accuracy_3_of_5‚0.022715052589774132 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974084496498108 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.044459111988544464 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.18529027700424194 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1875000149011612 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.23976537585258484 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3310035467147827 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08314498513936996 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0712575688958168 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17747658491134644 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12145086377859116 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9846881031990051 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9892882108688354 (particle/f_score‚0.32562750577926636 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0015403786674141884 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1535351276397705 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.058942943811416626 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0042041935957968235 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.39592769742012024 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3003738224506378 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.16953827440738678 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2507440447807312 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13500206172466278 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16192756593227386 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.5643094778060913 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957049489021301 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9761378765106201 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.20306149125099182 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.17660818994045258 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983565807342529 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.29497578740119934 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12536588311195374 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.17995087802410126 ("Purity/3FRt2SRqqt1FBt/event_purit¤ ñŽ y‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.12155557423830032 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.09182099252939224 particle/accuracy_2_of_6‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2679619789123535 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.11481481045484543 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.4545454680919647 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17777778208255768 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.4444444477558136 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.27287065982818604 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.18505747616291046 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.21976040303707123 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17041446268558502 particle/accuracy_1_of_4‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.1881113350391388 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.18700920045375824 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.983421802520752 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.21016374230384827 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 (particle/accuracy‚0.8336178660392761/yudu› ¤¥ =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.1739978790283203 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9648424983024597 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9794096946716309 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.15419742465019226 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05525325983762741 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.18917907774448395 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08067916333675385 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.27609676122665405 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.25438594818115234 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07094291597604752 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.15330331027507782 particle/accuracy_3_of_6‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.29999998211860657 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.19711525738239288 particle/accuracy_1_of_5‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 particle/accuracy_4_of_7‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.285849928855896 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2576594948768616 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3184027373790741 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13374485075473785 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.0943983793258667 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1666063368320465 ,jet/accuracy_3_of_4‚0.026402678340673447 particle/accuracy_2_of_2‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17555353045463562 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8155224323272705 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.23598404228687286 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.23863635957241058 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_2_of_7‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.33024829626083374 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.115079365670681 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16974475979804993 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.16106584668159485 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9973545670509338 particle/accuracy_3_of_5‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9792776107788086 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.001799962599761784 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995074450969696 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1449326127767563 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.05000000074505806 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.2199999988079071 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.14413894712924957 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18097653985023499 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1481473445892334 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.983563244342804 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.16341683268547058 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0011337868636474013 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.23076923191547394 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1814814805984497 particle/accuracy_2_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.956250011920929 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.22376833856105804 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.2646174132823944 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17882905900478363 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11724557727575302 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_3_of_4‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1748754382133484 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9793145656585693 jet/accuracy_4_of_5‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.23333333432674408 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1458333283662796 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16638554632663727 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚1208 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_1‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2994663715362549 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.2857142984867096 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1458333283662796 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974084496498108 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.15603460371494293 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.17298851907253265 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0028656101785600185 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.23895929753780365 +jet/accuracy_2_of_3‚0.14665542542934418 particle/accuracy_7_of_7‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32604196667671204 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.014367816969752312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.059696830809116364 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.18774832785129547 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.09606862813234329 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1527777761220932 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07094291597604752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.17777778208255768 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13409961760044098 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.15229886770248413 particle/accuracy_4_of_4‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.31712964177131653 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.34266993403434753 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.17002059519290924 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.4545454680919647 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3296545743942261 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13733887672424316 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.44999998807907104 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.7616811394691467 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.007577079813927412 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.18891634047031403 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2705555260181427 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.1600281149148941 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.23534482717514038 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9953703880310059 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9852305054664612 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.17803767323493958 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9954648613929749 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17335672676563263 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.16049383580684662 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.17418727278709412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2571648955345154 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.19033953547477722 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.3125 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3513636887073517 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1547335535287857 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6113747358322144 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03061659261584282 )jet/accuracy_1_of_1‚0.308608740568161 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2425837367773056 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.19190643727779388 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2589065134525299 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.38017043471336365 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.13271604478359222 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.17520484328269958 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9402198195457458 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12932829558849335 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2757936716079712 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18344224989414215 & Purity/2FRt0SRqqt3FBt/FRt_purity‚1 % _timestamp‚1.7884058446391487e+09 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 particle/accuracy_5_of_7‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19051723182201385 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02380952425301075 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_2_of_5‚0.29191628098487854 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.13700349628925323 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.1590360552072525 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003968254197388887 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.2857142984867096 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2757367193698883 jet/accuracy_3_of_7‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0022694868966937065 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.21827943623065948 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.21733863651752472 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9875777959823608 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1453070342540741 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1264546811580658 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.10330472886562347 *jet/accuracy_1_of_6‚0.8269230723381042 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_4_of_4‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_6‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9889121651649475 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.1805555522441864 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 epoch‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9843008518218994 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.24321705102920532 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16653752326965332 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1666666567325592 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9885929226875305 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.19361571967601776 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.009408602491021156 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1619369387626648 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.3947368562221527 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.053030308336019516 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2321428507566452 jet/accuracy_7_of_7‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9314420819282532 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1636904776096344 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_5‚0.8062660694122314 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.04888888821005821 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.22070260345935822 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.992909848690033 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15543650090694427 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.15603460371494293 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.01259425189346075 ,jet/accuracy_3_of_5‚0.022715052589774132 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974084496498108 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.044459111988544464 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.18529027700424194 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1875000149011612 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.23976537585258484 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3310035467147827 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08314498513936996 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0712575688958168 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17747658491134644 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12145086377859116 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9846881031990051 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9892882108688354 (particle/f_score‚0.32562750577926636 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0015403786674141884 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1535351276397705 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.058942943811416626 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0042041935957968235 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.39592769742012024 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3003738224506378 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.16953827440738678 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2507440447807312 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13500206172466278 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16192756593227386 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.5643094778060913 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957049489021301 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9761378765106201 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.20306149125099182 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.17660818994045258 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983565807342529 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.29497578740119934 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12536588311195374 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.17995087802410126 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.12155557423830032 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.09182099252939224 particle/accuracy_2_of_6‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2679619789123535 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.11481481045484543 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.4545454680919647 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17777778208255768 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.4444444477558136 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.27287065982818604 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.18505747616291046 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.21976040303707123 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17041446268558502 particle/accuracy_1_of_4‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.1881113350391388 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.18700920045375824 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.983421802520752 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.21016374230384827 ?&PuriB1U²Gty/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 (particle/accuracy‚0.8336178660392761 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.08096494525671005 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.32225272059440613 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9858579039573669 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.094040147960186 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.166173055768013 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16007453203201294 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 )loss/val_total_loss‚64.30039961368428 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.14259260892868042 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9945833086967468 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.05000000074505806 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9963295459747314 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9933606386184692 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.36682358384132385 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6465815305709839 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 jet/accuracy_6_of_6‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.18825297057628632 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.29333335161209106 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938377737998962 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2519345283508301 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.012975560501217842 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18859648704528809 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.30184027552604675 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2857142984867096 _runtime‚ 947.628535002 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.2574055790901184 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05728926509618759 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07603643089532852 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.6954002380371094 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9937106966972351 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.004036904312670231 jet/accuracy_5_of_5‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.06666667014360428 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9924932718276978 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_3_of_6‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3128848671913147 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 particle/accuracy_1_of_7‚1 particle/accuracy_3_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.005807508248835802 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 particle/accuracy_6_of_7‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 +jet/accuracy_2_of_2‚0.06767892837524414 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07230252027511597 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1751020848751068 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009876543655991554 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 _step‚98b8YÐ’œ ‚(:‡ áÐãÔ„†„  gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Î‘æI ‚(:¾ áÐãÔÙÖÌ  network.sent‚ 1.591119e+06 disk.sda1.in‚0 disk.nvme0n1.in‚8.34375 network.recv‚408914" disk.dm-0.out‚3264.17626953125 disk.sda.out‚ 47.08203125 disk.drbd2080.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚3264.17626953125 disk.rbd1.out‚ 4.9423828125% disk.nvme0n1.out‚3264.17626953125* disk./.usagePercent‚15.912292422947058 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0$ memory_percent‚94.14920806884766# proc.memory.rssMB‚ 2814.40234375 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd1.in‚0 cpu‚25.36916974189899 disk.dm-0.in‚8.34375 disk.drbd2265.in‚0 disk.sda3.out‚ 47.08203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0* proc.memory.percent‚17.177748680114746 disk.drbd1716.out‚0 disk.sda.in‚0 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375 disk.sda3.in‚0% disk./.usageGB‚237.10465621948242 disk.dm-1.out‚ 47.08203125 disk.sda1.out‚0% proc.memory.availableMB‚ 958.59375õ¹n’ž ‚(:‡ ðÐãÔŸé• gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷8…“HŸ ‚(:½ ðÐãÔ˳  disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 47.47265625 disk.sda3.out‚ 47.47265625 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 5.0908203125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚8.34375 disk.dm-1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2814.40234375% disk./.usageGB‚237.10465621948242 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0$ memory_percent‚94.15016174316406$ proc.memory.availableMB‚958.4375* proc.memory.percent‚17.177748680114746 cpu‚25.364195019155186' disk.nvme0c0n1.out‚3264.37939453125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚8.34375 network.recv‚414387* disk./.usagePercent‚15.912292422947058 disk.drbd2265.in‚0 disk.sda.out‚ 47.47265625 disk.drbd2080.in‚0 disk.sda2.out‚0 network.sent‚ 1.71384e+06% disk.nvme0n1.out‚3264.37939453125 disk.dm-0.in‚8.34375" disk.dm-0.out‚3264.37939453125aÝgÌ0  + _wandbruntime‚981  _runtime‚981£»¹Æ¡ À _step‚99 ' lr-AdamW/pg1‚0.00027504987825675576 ' lr-AdamW/pg2‚0.00027504987825675576 trainer/global_step‚1249 $ _timestamp‚1.788405878180843e+09 _runtime‚ 981.168351818à3åÊ¢ Ä ' lr-AdamW/pg1‚0.00027504987825675576 ' lr-AdamW/pg2‚0.00027504987825675576 trainer/global_step‚1249 $ _timestamp‚1.788405878180843e+09 _runtime‚ 981.168351818 _step‚99cÎÀìw0£ + _wandbruntime‚981  _runtime‚981ø„“µÆ'¤ ÀO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73826573425152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23627720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6075594 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79470224662016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1044210331391 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70869353436160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79470248602112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72778502860288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1047414203392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18208374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1101410504192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 981.169249832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10625454 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884058781815882e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚100 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72778484857344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80571635166208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚73825899060736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6076020 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1043555893643 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23607684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16701474 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18208376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048070564864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18208376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157333504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72778502860288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10625452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1101567837696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41816058 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73825899060736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72778484857344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048070564864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73825899060736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72778484857344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048070564864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6075594 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79470224662016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1044210331391 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70869353436160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79470248602112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72778502860288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1047414203392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18208374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1101410504192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 981.169249832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10625454 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884058781815882e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚100 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72778484857344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80571635166208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚73825899060736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6076020 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1043555893643 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23607684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16701474 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18208376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048070564864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18208376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157333504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72778502860288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10625452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1101567837696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430d~à`0¦ + _wandbruntime‚981  _runtime‚981•ý\`ù§ ÊM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71968629895327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18222392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80634332342272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72834837896192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1048233631232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41848528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70924274606080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 1019úE×!4255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884058788763945e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1048233631232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23626138 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1045026766563 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16713703 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79531899339776 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72834855899136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10633620 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16714141 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10633622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71969301372643 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72834855899136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161514496 _step‚101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80634146887680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1102433002496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 981.864395391 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚1249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6080083 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79531875399680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73883071527424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1048890006016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1044372328607 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18222390 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18222390 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23646200 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73883745905152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73883745905152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18222392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6080519(»Ô&¨ ÎM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73883071527424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1048890006016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1044372328607 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18222390 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18222390 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23646200 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73883745905152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73883745905152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18222392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6080519 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71968629895327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18222392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80634332342272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72834837896192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1048233631232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41848528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70924274606080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884058788763945e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1048233631232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23626138 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1045026766563 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16713703 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79531899339776 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72834855899136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10633620 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16714141 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10633622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71969301372643 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72834855899136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161514496 _step‚101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80634146887680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1102433002496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 981.864395391 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚1249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6080083 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79531875399680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0e®4X0© + _wandbruntime‚981  _runtime‚981ûÓBySª Í 1loss/SRqqt4/assignment_loss‚2.057693983594032 .loss/FRt2/detection_loss‚2.539957594871521 1loss/SRqqt1/detection_loss‚2.1981702491641046 _step‚102 /loss/FRt1/assignment_loss‚4.672743744996129 /loss/FRt2/assignment_loss‚5.193051195144653 0loss/FBt4/assignment_loss‚0.9121381103992462 /loss/FRt4/detection_loss‚2.2976758383652744 .loss/FRt3/assignment_loss‚3.32289927249605 1loss/SRqqt1/assignment_loss‚2.051492393016815 1loss/FBt1/assignment_loss‚0.46397775914520023 .loss/FBt4/detection_loss‚4.524711990356446 _runtime‚ 981.865864582 1loss/SRqqt2/detection_loss‚3.0093530753384465 1loss/SRqqt4/detection_loss‚1.7554847289846367 /loss/FBt1/detection_loss‚3.7748923152685165 %loss/total_loss‚58.03323630903211 trainer/global_step‚1249 0loss/FRt4/assignment_loss‚3.8731303495519303 /loss/FRt1/detection_loss‚2.2927688469692153 .loss/FBt2/detection_loss‚2.268683416148027 /loss/FBt3/detection_loss‚1.1496878961030765 1loss/SRqqt2/assignment_loss‚2.884854617326156 2loss/SRqqt3/assignment_loss‚1.7318189917132258 0loss/FBt2/assignment_loss‚0.5681213527193808 /loss/FBt3/assignment_loss‚0.301104482819755 .loss/FRt3/detection_loss‚2.354765507477251 1loss/SRqqt3/detection_loss‚1.8340585970630248 epoch‚1 % _timestamp‚1.7884058788784773e+09ó×â/W« Ñ .loss/FRt3/assignment_loss‚3.32289927249605 1loss/SRqqt1/assignment_loss‚2.051492393016815 1loss/FBt1/assignment_loss‚0.46397775914520023 .loss/FBt4/detection_loss‚4.524711990356446 _runtime‚ 981.865864582 1loss/SRqqt2/detection_loss‚3.0093530753384465 1loss/SRqqt4/detection_loss‚1.7554847289846367 /loss/FBt1/detection_loss‚3.7748923152685165 %loss/total_loss‚58.03323630903211 trainer/global_step‚1249 0loss/FRt4/assignment_loss‚3.8731303495519303 /loss/FRt1/detection_loss‚2.2927688469692153 .loss/FBt2/detection_loss‚2.268683416148027 /loss/FBt3/detection_loss‚1.1496878961030765 1loss/SRqqt2/assignment_loss‚2.884854617326156 2loss/SRqqt3/assignment_loss‚1.7318189917132258 0loss/FBt2/assignment_loss‚0.5681213527193808 /loss/FBt3/assignment_loss‚0.301104482819755 .loss/FRt3/detection_loss‚2.354765507477251 1loss/SRqqt3/detection_loss‚1.8340585970630248 epoch‚1 % _timestamp‚1.7884058788784773e+09 1loss/SRqqt4/assignment_loss‚2.057693983594032 .loss/FRt2/detection_loss‚2.539957594871521 1loss/SRqqt1/detection_loss‚2.1981702491641046 _step‚102 /loss/FRt1/assignment_loss‚4.672743744996129 /loss/FRt2/assignment_loss‚5.193051195144653 0loss/FBt4/assignment_loss‚0.9121381103992462 /loss/FRt4/detection_loss‚2.2976758383652744fªò’¬ ‚(:‡ ÿÐãÔ£÷¢ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êg<ƒE­ ‚(:º ÿÐãÔ§Ýè  network.recv‚418505) disk./.usagePercent‚15.91217184339843 disk.drbd1716.in‚0 disk.dm-0.in‚8.34375 disk.dm-0.out‚3739.146484375 disk.nvme0c0n1.in‚8.34375 disk.drbd2265.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2814.40234375 disk.dm-1.out‚ 48.10546875 disk.sda.in‚0$ disk./.usageGB‚237.1028594970703 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0$ memory_percent‚94.15395259857178( proc.memory.availableMB‚ 957.81640625 disk.sda3.out‚ 48.10546875* proc.memory.percent‚17.177748680114746 cpu‚25.359828604334044 network.sent‚ 1.719857e+06% disk.nvme0c0n1.out‚3739.146484375 disk.nvme0n1.in‚8.34375# disk.nvme0n1.out‚3739.146484375 disk.sda.out‚ 48.10546875 disk.drbd1716.out‚0 disk.rbd1.out‚ 5.0908203125 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0ƒ’©;® ‚(:… ŽÑãÔ¬ƒŠ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚146.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûý×F¯ ‚(:» ŽÑãÔ“Û§  disk.dm-0.in‚8.34375 disk.dm-1.out‚ 48.62109375 disk.drbd1716.out‚0 proc.cpu.threads‚17 network.recv‚424781 disk.dm-0.out‚3739.373046875 disk.drbd2265.in‚0 disk.sda.out‚ 48.62109375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚8.34375 disk.sda2.out‚0% disk.nvme0c0n1.out‚3739.373046875* proc.memory.percent‚17.177748680114746 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 5.3486328125 disk.sda.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 network.sent‚ 1.828068e+06% disk./.usageGB‚237.10290145874023# memory_percent‚94.1537618637085 disk.sda3.out‚ 48.62109375 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375# disk.nvme0n1.out‚3739.373046875# proc.memory.rssMB‚ 2814.40234375 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 957.84765625 cpu‚25.355240797809156* disk./.usagePercent‚15.9121746594813072ö2° - _wandbruntime‚1017  _runtime‚1017 s¿ïɱ à trainer/global_step‚1299 % _timestamp‚1.7884059145919514e+09 _runtime‚1017.579419475 _step‚103 ' lr-AdamW/pg1‚0.00027501262088939707 ' lr-AdamW/pg2‚0.00027501262088939707Ù•]Ͳ Ç ' lr-AdamW/pg1‚0.00027501262088939707 ' lr-AdamW/pg2‚0.00027501262088939707 trainer/global_step‚1299 % _timestamp‚1.7884059145919514e+09 _runtime‚1017.579419475 _step‚103guEo]2³ - _wandbruntime‚1017  _runtime‚1017k»¨r´ ÀO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74698951197211 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18909174 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83697240020480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75596136799744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24533736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83697050358272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82552785485312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73614577003520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1089056370688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76684536808960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1144454535168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24533736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75596136799744 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1089056370688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11033854 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freeÕ|‹T d‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚1299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18909174 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83697050358272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82552785485312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73614577003520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1089056370688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76684536808960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1144454535168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24533736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75596136799744 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1089056370688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11033854 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚1299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18909174 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74698951197211 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18909174 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83697240020480hY±än2¶ - _wandbruntime‚1018  _runtime‚1018=†´Ñ&· ËM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11042020 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82614412282880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1145318011392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73669464094720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1085207668015 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24572252 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73669481134080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43475380 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75652507841536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75652489838592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1089219437056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83759754234368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18923190 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18923192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18923190 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 trainer/global_step‚1299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159417344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24552190 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82614436222976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75652507841536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1089875811840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74754671762735 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17348513 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43495444 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚75652489838592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6306493 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24552190 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76741709275648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24572252 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74755343240051 _step‚105 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76742383653376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6306917 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1089875811840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1145158594048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1085862105971 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1089219437056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884059152789376e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0M¿®!Õ&¸ ÏM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11042020 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82614412282880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1145318011392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73669464094720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1085207668015 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24572252 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73669481134080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43475380 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75652507841536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75652489838592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1089219437056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83759754234368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18923190 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18923192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18923190 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 trainer/global_step‚1299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159417344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24552190 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82614436222976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75652507841536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1089875811840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74754671762735 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17348513 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43495444 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚75652489838592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6306493 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24552190 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76741709275648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24572252 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74755343240051 _step‚105 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76742383653376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6306917 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1089875811840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1145158594048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1085862105971 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1089219437056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884059152789376e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944i¼iÕ2¹ - _wandbruntime‚1018  _runtime‚1018$¹\§º Á 2loss/SRqqt1/assignment_loss‚2.1059620953523197 0loss/FBt2/assignment_loss‚0.5535613099733953 loss/FBt4/assignment_loss‚0 .loss/FBt4/detection_lod(% ss‚2.209042549133301 /loss/FRt4/detection_loss‚2.2076115376990417 .loss/FBt2/detection_loss‚3.347653212646643 /loss/FBt3/detection_loss‚0.9340079671913234 $loss/total_loss‚54.7319301330443 _step‚106 /loss/FRt3/assignment_loss‚2.868084888006079 /loss/FRt1/detection_loss‚2.7166577787012667 .loss/FRt2/detection_loss‚2.717170497466778 1loss/SRqqt4/detection_loss‚1.9321682551313961 0loss/FRt1/assignment_loss‚3.7874204758051278 /loss/FRt4/assignment_loss‚4.261310515732601 % _timestamp‚1.7884059152809284e+09 2loss/SRqqt2/assignment_loss‚2.7485313568359766 1loss/SRqqt2/detection_loss‚2.4213190720631528 epoch‚1 _runtime‚1018.268267808 trainer/global_step‚1299 1loss/SRqqt3/assignment_loss‚2.101174307987094 2loss/SRqqt4/assignment_loss‚1.6934907240053965 1loss/FBt3/assignment_loss‚0.22377339543543473 /loss/FBt1/detection_loss‚4.1679299260888785 /loss/FRt3/detection_loss‚1.8896861728923073 1loss/SRqqt1/detection_loss‚2.1545941514464526 1loss/SRqqt3/detection_loss‚1.9294900325841682 /loss/FRt2/assignment_loss‚5.171155354072308 0loss/FBt1/assignment_loss‚0.5901345567938604ùà„ÀK» Å $loss/total_loss‚54.7319301330443 _step‚106 /loss/FRt3/assignment_loss‚2.868084888006079 /loss/FRt1/detection_loss‚2.7166577787012667 .loss/FRt2/detection_loss‚2.717170497466778 1loss/SRqqt4/detection_loss‚1.9321682551313961 0loss/FRt1/assignment_loss‚3.7874204758051278 /loss/FRt4/assignment_loss‚4.261310515732601 % _timestamp‚1.7884059152809284e+09 2loss/SRqqt2/assignment_loss‚2.7485313568359766 1loss/SRqqt2/detection_loss‚2.4213190720631528 epoch‚1 _runtime‚1018.268267808 trainer/global_step‚1299 1loss/SRqqt3/assignment_loss‚2.101174307987094 2loss/SRqqt4/assignment_loss‚1.6934907240053965 1loss/FBt3/assignment_loss‚0.22377339543543473 /loss/FBt1/detection_loss‚4.1679299260888785 /loss/FRt3/detection_loss‚1.8896861728923073 1loss/SRqqt1/detection_loss‚2.1545941514464526 1loss/SRqqt3/detection_loss‚1.9294900325841682 /loss/FRt2/assignment_loss‚5.171155354072308 0loss/FBt1/assignment_loss‚0.5901345567938604 2loss/SRqqt1/assignment_loss‚2.1059620953523197 0loss/FBt2/assignment_loss‚0.5535613099733953 loss/FBt4/assignment_loss‚0 .loss/FBt4/detection_loss‚2.209042549133301 /loss/FRt4/detection_loss‚2.2076115376990417 .loss/FBt2/detection_loss‚3.347653212646643 /loss/FBt3/detection_loss‚0.9340079671913234jNJ瑼 ‚(:† ÑãÔ½‚’ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚185.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R…jL½ ‚(:Á ÑãÔ˜¼ó $ memory_percent‚94.15688514709473 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375 disk.sda.out‚ 49.50390625 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.in‚0* disk./.usagePercent‚15.912170819368294 disk.rbd1.out‚ 5.3486328125 disk.dm-0.in‚8.34375* proc.memory.percent‚17.177748680114746 network.sent‚ 1.834417e+06 disk.sda3.out‚ 49.50390625 disk.drbd1716.out‚0" disk.dm-0.out‚3740.33837890625 disk.drbd2265.in‚0 disk.sda.in‚0 disk.drbd2080.in‚0% disk./.usageGB‚237.10284423828125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 cpu‚25.350640712702972 disk.drbd1716.in‚0% disk.nvme0n1.out‚3740.33837890625 disk.dm-1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 49.50390625 disk.drbd2265.out‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚3740.33837890625' proc.memory.availableMB‚ 957.3359375# proc.memory.rssMB‚ 2814.40234375 network.recv‚429296 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚8.34375Øk$y’¾ ‚(:‡ ¬ÑãÔÕÌ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.644' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3288 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HG‹L¿ ‚(:Á ¬ÑãÔ•Åí  disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375$ memory_percent‚94.15490627288818* proc.memory.percent‚17.177748680114746 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 957.66015625# proc.memory.rssMB‚ 2814.40234375 proc.cpu.threads‚17 disk.sda1.out‚0" disk.dm-0.out‚3741.01025390625 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 50.13671875 disk.drbd2080.in‚0' disk.nvme0c0n1.out‚3741.01025390625* disk./.usagePercent‚15.912278342532673 disk.rbd0.in‚0 disk.rbd1.out‚ 5.4111328125 network.sent‚ 1.941839e+06 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd1716.in‚0 disk.dm-0.in‚8.34375 network.recv‚434842 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚3741.01025390625 cpu‚25.346202406636376$ disk./.usageGB‚237.1044464111328 disk.sda.out‚ 50.13671875 disk.sda3.out‚ 50.13671875’•8¾À ‚(:… »ÑãÔÍÓº gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D8¸GGÁ ‚(:¼ »ÑãÔ±ìå  disk.sda1.in‚0 disk.sda3.out‚ 50.76953125 cpu‚25.342072309000958! disk.dm-0.out‚3741.4541015625 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚8.34375 disk.nvme0n1.in‚8.34375 network.sent‚ 1.947823e+06$ disk./.usageGB‚237.1044807434082 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 50.76953125* disk./.usagePercent‚15.912280646600482 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚8.34375 disk.dm-1.in‚0 disk.drbd2265.in‚0 network.recv‚438953 disk.sda2.out‚0& disk.nvme0c0n1.out‚3741.4541015625 disk.sda.out‚ 50.76953125# proc.memory.rssMB‚ 2814.40234375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 5.4111328125$ disk.nvme0n1.out‚3741.4541015625 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚94.15502548217773& proc.memory.availableMB‚ 957.640625* proc.memory.percent‚17.177748680114746 proc.cpu.threads‚17X®ßï2 - _wandbruntime‚1054  _runtime‚1054"ŒlÇà Á trainer/global_step‚1349 % _timestamp‚1.7884059513065012e+09 _runtime‚ 1054.2939622 _step‚107 ' lr-AdamW/pg1‚0.00027494671225090054 ' lr-AdamW/pg2‚0.00027494671225090054îH™˜ËÄ Å ' lr-AdamW/pg1‚0.00027494671225090054 ' lr-AdamW/pg2‚0.00027494671225090054 trainer/global_step‚1349 % _timestamp‚1.7884059513065012e+09 _runtime‚ 1054.2939622 _step‚107k£oµõ2Å - _wandbruntime‚1054  _runtime‚1054ˆoÇ'Æ ÁO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25459788 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79543174557184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78413788742144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78413806745088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 % _timestamp‚1.7884059513072326e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77485664541727 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17971989 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1125226572459 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1130042176512 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76359783531520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6529735 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1129385815040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85635322368512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11442252 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1130042176512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19609974 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79543848921600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78413806745088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76359766492160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1187113669120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78413788742144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6529308 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45089800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚108 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19609974 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45069762 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77484993064619 _runtime‚1054.2949848095wË'Ç ÅO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1187113669120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78413788742144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6529308 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45089800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚108 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19609974 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45069762 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77484993064619 _runtime‚1054.294984809 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25459788 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79543174557184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78413788742144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78413806745088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 % _timestamp‚1.7884059513072326e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77485664541727 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17971989 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1125226572459 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1130042176512 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76359783531520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6529735 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1129385815040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85635322368512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11442252 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1130042176512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19609974 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79543848921600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78413806745088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76359766492160l!›ù‹2È - _wandbruntime‚1054  _runtime‚1054úÚ1îYÉ ËM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79601021401600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86884923999744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1187974833664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76414687662080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19623992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25478242 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78470159783936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45102232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45102232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78470141780992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11450420 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1126697445379 % _timestamp‚1.7884059519921842e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85696949166080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77540713630143 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25498304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1130205242880 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79600347023872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78470141780992 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85696973106176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1188138445312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1130205242880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚109 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1126043007423 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚1349 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86885111551488 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77541385107459 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6533828 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1054.980014998 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78470159783936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19623990 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45122296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17984693 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1130861617664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19623990 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17984248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656OÿÕ&Ê ÏM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6533828 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1054.980014998 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78470159783936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19623990 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45122296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17984693 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1130861617664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19623990 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17984248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79601021401600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86884923999744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1187974833664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76414687662080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19623992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25478242 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78470159783936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45102232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45102232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78470141780992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11450420 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1126697445379 % _timestamp‚1.7884059519921842e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85696949166080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77540713630143 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25498304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1130205242880 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79600347023872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78470141780992 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85696973106176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1188138445312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1130205242880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚109 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1126043007423 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚1349 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86885111551488 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77541385107459 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944mUì`Á2Ë - _wandbruntime‚1054  _runtime‚1054ö J9XÌ Ò .loss/FRt4/detection_loss‚2.510554859584028 0loss/SRqqt1/detection_loss‚2.455792801409233 &loss/total_loss‚54.831791668267016 1loss/SRqqt2/detection_loss‚2.0747028939425944 epoch‚1 _step‚110 .loss/FRt2/assignment_loss‚4.89007500446204 1loss/SRqqt3/assignment_loss‚2.181084742159422 /loss/FBt1/detection_loss‚3.5589833319187165 _runtime‚1054.981120825 0loss/FBt3/assignment_loss‚0.2999100894261261 .loss/FRt3/detection_loss‚1.983340545612223 1loss/SRqqt3/detection_loss‚1.4485436523782795 0loss/FRt1/assignment_loss‚3.8623752453747917 0loss/FRt3/assignment_loss‚2.1372878352801004 /loss/FRt1/detection_loss‚2.8404335923054638 1loss/SRqqt4/detection_loss‚1.9542932099983341 /loss/FBt2/detection_loss‚2.2393812850117683 2loss/SRqqt4/assignment_loss‚1.9292447939515114 0loss/FBt4/assignment_loss‚1.2124785166233778 trainer/global_step‚1349 % _timestamp‚1.7884059519937503e+09 2loss/SRqqt2/assignment_loss‚2.2774168265983463 /loss/FBt2/assignment_loss‚0.758696953952304 /loss/FBt3/detection_loss‚0.8983613703337971 /loss/FBt4/detection_loss‚3.7277116179466248 1loss/SRqqt1/assignment_loss‚2.179484783149347 1loss/FBt1/assignment_loss‚0.31317576648143586 0loss/FRt4/assignment_loss‚4.5985125763849775 .loss/FRt2/detection_loss‚2.499949373982169·ÐǦ\Í Ö 1loss/SRqqt2/detection_loss‚2.0747028939425944 epoch‚1 _step‚110 .loss/FRt2/assignment_loss‚4.89007500446204 1loss/SRqqt3/assignment_loss‚2.181084742159422 /loss/FBt1/detection_loss‚3.5589833319187165 _runtime‚1054.981120825 0loss/FBt3/assignment_loss‚0.2999100894261261 .loss/FRt3/detection_loss‚1.983340545612223 1loss/SRqqt3/detection_loss‚1.4485436523782795 0loss/FRt1/assignment_loss‚3.8623752453747917 0loss/FRt3/assignment_loss‚2.1372878352801004 /loss/FRt1/detection_loss‚2.8404335923054638 1loss/SRqqt4/detection_loss‚1.9542932099983341 /loss/FBt2/detection_loss‚2.2393812850117683 2loss/SRqqt4/assignment_loss‚1.9292447939515114 0loss/FBt4/assignment_loss‚1.2124785166233778 trainer/global_step‚1349 % _timestamp‚1.7884059519937503e+09 2loss/SRqqt2/assignment_loss‚2.2774168265983463 /loss/FBt2/assignment_loss‚0.758696953952304 /loss/FBt3/detection_loss‚0.8983613703337971 /loss/FBt4/detection_loss‚3.7277116179466248 1loss/SRqqt1/assignment_loss‚2.179484783149347 1loss/FBt1/assignment_loss‚0.31317576648143586 0loss/FRt4/assignment_loss‚4.5985125763849775 .loss/FRt2/detection_loss‚2.499949373982169 .loss/FRt4/detection_loss‚2.510554859584028 0loss/SRqqt1/detection_loss‚2.455792801409233 &loss/total_loss‚54.831791668267016nã¯E’Î ‚(:‡ ÊÑãÔ˲° gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.244' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05i<áHÏ ‚(:½ ÊÑãÔ»ÓŸ  proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0$ memory_percent‚94.15791034698486 disk.sda1.out‚0 disk.dm-0.in‚8.34375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda3.out‚ 51.21484375 disk.rbd1.in‚0 cpu‚25.33798313590991$ disk.nvme0n1.out‚3741.6884765625 network.sent‚ 1.953605e+06 disk.drbd1716.in‚0 disk.nvme0n1.in‚8.34375! disk.dm-0.out‚3741.6884765625 disk.sda.out‚ 51.21484375 disk.sda3.in‚0* disk./.usagePercent‚15.912235077259387 disk.rbd0.in‚0 disk.sda1.in‚0% disk./.usageGB‚237.10380172729492 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375& disk.nvme0c0n1.out‚3741.6884765625 disk.dm-1.in‚0( proc.memory.availableMB‚ 957.16796875 network.recv‚443513 disk.rbd1.out‚ 5.552734375 disk.dm-1.out‚ 51.21484375 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2814.40234375* proc.memory.percent‚17.1777486801147468Q% ‘Ð ‚(:† ÙÑãÔÂϺ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.325' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.065 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06˜2sBÑ ‚(:· ÙÑãÔÜŽã  disk.rbd0.in‚0 disk.nvme0c0n1.in‚8.34375$ memory_percent‚94.15872097015381( proc.memory.availableMB‚ 957.03515625* proc.memory.percent‚17.177748680114746 disk.dm-1.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0% disk./.usageGB‚237.10374450683594 disk.drbd2265.in‚0 disk.sda3.out‚ 51.734375 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2814.40234375 cpu‚25.334144073569526 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 51.734375 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 5.552734375 network.sent‚ 2.06208e+06 disk.dm-0.in‚8.34375 disk.drbd2266.out‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚15.912231237146374 disk.sda.in‚0 proc.cpu.threads‚17& disk.nvme0c0n1.out‚3742.6669921875 network.recv‚449821 disk.nvme0n1.in‚8.34375$ disk.nvme0n1.out‚3742.6669921875! disk.dm-0.out‚3742.6669921875 disk.drbd1716.out‚0 disk.dm-1.out‚ 51.734375¤^¥2Ò - _wandbruntime‚1090  _runtime‚1090fÛÆÓ À % _timestamp‚1.7884059878460429e+09 _runtime‚ 1090.83347794 _step‚111 & lr-AdamW/pg1‚0.0002748521660765277 & lr-AdamW/pg2‚0.0002748521660765277 trainer/global_step‚1399=K¤JÊÔ Ä _step‚111 & lr-AdamW/pg1‚0.0002748521660765277 & lr-AdamW/pg2‚0.0002748521660765277 trainer/global_step‚1399 % _timestamp‚1.7884059878460429e+09 _runtime‚ 1090.83347794o_Þ4¿2Õ - _wandbruntime‚1090  _runtime‚1090 fÇ'Ö ÁO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11850652 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6756164 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82401812305408 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80271034932027 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1170371620864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1166061911867 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81231458687488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81231440684544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6756609 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79104973020160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82401812305408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82402486669824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26405876 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46716652 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20310776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20310776 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26385840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1090.834518443 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1171027982336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79104990059520 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81231458687488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11850654 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1230115038208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80271706409135 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884059878468318e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1170371620864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _step‚112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46696614 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26385840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18607263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187565056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26405876 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82402486669824ôÓŸ9ü× ÅO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82402486669824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26405876 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46716652 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20310776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20310776 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26385840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1090.834518443 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1171027982336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79104990059520 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81231458687488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11850654 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1230115038208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80271706409135 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884059878468318e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1170371620864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _step‚112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46696614 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26385840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18607263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187565056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26405876 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82402486669824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11850652 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6756164 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82401812305408 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80271034932027 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1170371620864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1166061911867 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81231458687488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81231440684544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6756609 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79104973020160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82401812305408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928p !Ó2Ø - _wandbruntime‚1091  _runtime‚1091ì®J‰Ñ&Ù ËM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26424356 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46749148 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81287811726336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90010283329024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 _runtime‚1091.521052756 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1171847423488 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6760631 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81287793723392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26424356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82459659149824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1166878346831 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1171191048704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18619896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46729084 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20324792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80327426974867 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80326755497551 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79159877150720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20324792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81287793723392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11858822 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79159894190080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81287811726336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46729084 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82459659149824 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6761074 % _timestamp‚1.7884059885332208e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1171847423488 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1230797279744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18619451 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20324790 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81287793723392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26424356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82459659149824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1166878346831 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1171191048704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18619896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46729084 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20324792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80327426974867 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80326755497551 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79159877150720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20324792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81287793723392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11858822 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79159894190080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81287811726336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46729084 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82459659149824 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6761074 % _timestamp‚1.7884059885332208e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1171847423488 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1230797279744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18619451 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20324790 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91800372194816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚116 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21011576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48343504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83057748276543 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81850179548160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84049110629888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93073290703872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 _runtime‚1127.128249335 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1212013788160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81850196587520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1212013788160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273088425472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85261124418048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12259052 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21011574 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1211357426688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6985431 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84049110629888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1207551689023 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884060241406326e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21011574 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48323466 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27311892 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12259054 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85260450053632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19244485 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83057076799435 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1272918509056 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206897251275R®j Ë'é ÅO % _timestamp‚1.7884060241406326e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21011574 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48323466 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27311892 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12259054 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85260450053632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19244485 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83057076799435 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1272918509056 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206897251275 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1211357426688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19244049 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21011576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27331928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6984997 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84049092626944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85260450053632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91800396134912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91800372194816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚116 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21011576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48343504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83057748276543 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81850179548160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84049110629888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93073290703872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 _runtime‚1127.128249335 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1212013788160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81850196587520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1212013788160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273088425472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85261124418048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12259052 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21011574 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1211357426688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6985431 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84049110629888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1207551689023 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0tåÚ¢2ê - _wandbruntime‚1127  _runtime‚1127¤[JÃë ËM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81905100718080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85317622520320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6989977 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84105463668736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 trainer/global_step‚1449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21025590 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93135794281472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1273771348992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27330346 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21025592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27350408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚1127.809154414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1208368124195 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21025592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85318296898048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85318296898048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84105445665792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48355936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93135986027520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚117 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83112797364959 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1212176854528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91862046872576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1207713686239 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21025590 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6989537 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81905083678720 % _timestamp‚1.7884060248211489e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12267220 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1212833229312 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48355936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated©8þ2‚84105463668736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1212176854528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19256757 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27330346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12267222 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19256757 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27330346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12267222 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81905100718080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85317622520320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6989977 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84105463668736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 trainer/global_step‚1449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21025590 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93135794281472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1273771348992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27330346 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21025592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27350408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚1127.809154414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1208368124195 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21025592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85318296898048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85318296898048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84105445665792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48355936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93135986027520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚117 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83112797364959 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1212176854528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91862046872576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1207713686239 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21025590 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6989537 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430u¾°¸2í - _wandbruntime‚1127  _runtime‚1127$UËYî Ó /loss/FRt3/assignment_loss‚2.359507936452116 1loss/SRqqt1/detection_loss‚2.5402226025859513 .loss/FBt2/detection_loss‚2.336107992074069 /loss/FBt3/detection_loss‚1.0409394269259689 /loss/FBt4/detection_loss‚4.7161301374435425 % _timestamp‚1.7884060248230093e+09 0loss/FBt1/assignment_loss‚0.4111828505918311 1loss/FBt3/assignment_loss‚0.24421252132086524 0loss/SRqqt4/detection_loss‚1.909996575996524 1loss/SRqqt3/assignment_loss‚1.627508359029889 1loss/SRqqt4/assignment_loss‚2.022092069270181 /loss/FRt3/detection_loss‚2.0435163054083074 1loss/SRqqt2/detection_loss‚2.2133453689716958 0loss/FRt1/assignment_loss‚3.8060016132170156 0loss/FRt4/assignment_loss‚4.2140319262232095 /loss/FRt1/detection_loss‚2.8660987269493843 0loss/FRt2/assignment_loss‚4.4658110499382015 2loss/SRqqt1/assignment_loss‚2.0700345846513906 /loss/FRt2/detection_loss‚2.3311335325241087 0loss/SRqqt3/detection_loss‚1.359932161512829 /loss/FBt1/detection_loss‚3.6723766922950745 %loss/total_loss‚53.85040872605123 _runtime‚ 1127.81038211 2loss/SRqqt2/assignment_loss‚2.3565814712563076 0loss/FBt2/assignment_loss‚0.6599972765244703 0loss/FBt4/assignment_loss‚0.3089345091393625 /loss/FRt4/detection_loss‚2.2747130357489294 trainer/global_step‚1449 epoch‚1 _step‚118cªµˆ]ï × 0loss/FRt1/assignment_loss‚3.8060016132170156 0loss/FRt4/assignment_loss‚4.2140319262232095 /loss/FRt1/detection_loss‚2.8660987269493843 0loss/FRt2/assignment_loss‚4.4658110499382015 2loss/SRqqt1/assignment_loss‚2.0700345846513906 /loss/FRt2/detection_loss‚2.3311335325241087 0loss/SRqqt3/detection_loss‚1.359932161512829 /loss/FBt1/detection_loss‚3.6723766922950745 %loss/total_loss‚53.85040872605123 _runtime‚ 1127.81038211 2loss/SRqqt2/assignment_loss‚2.3565814712563076 0loss/FBt2/assignment_loss‚0.6599972765244703 0loss/FBt4/assignment_loss‚0.3089345091393625 /loss/FRt4/detection_loss‚2.2747130357489294 trainer/global_step‚1449 epoch‚1 _step‚118 /loss/FRt3/assignment_loss‚2.359507936452116 1loss/SRqqt1/detection_loss‚2.5402226025859513 .loss/FBt2/detection_loss‚2.336107992074069 /loss/FBt3/detection_loss‚1.0409394269259689 /loss/FBt4/detection_loss‚4.7161301374435425 % _timestamp‚1.7884060248230093e+09 0loss/FBt1/assignment_loss‚0.4111828505918311 1loss/FBt3/assignment_loss‚0.24421252132086524 0loss/SRqqt4/detection_loss‚1.909996575996524 1loss/SRqqt3/assignment_loss‚1.627508359029889 1loss/SRqqt4/assignment_loss‚2.022092069270181 /loss/FRt3/detection_loss‚2.0435163054083074 1loss/SRqqt2/detection_loss‚2.2133453689716958vgE’ð ‚(:‡ •ÒãÔú»± gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚157.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~öíLñ ‚(:Á •ÒãÔó§  disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚8.34375 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 5.7568359375 disk.drbd2265.in‚0% disk./.usageGB‚237.10429000854492 disk.sda3.in‚0' disk.nvme0c0n1.out‚3745.08935546875 disk.drbd1716.in‚0% disk.nvme0n1.out‚3745.08935546875( proc.memory.availableMB‚ 955.83203125 disk.nvme0c0n1.in‚8.34375 cpu‚25.319494269137085) disk./.usagePercent‚15.91226784622377 disk.sda1.out‚0" disk.dm-0.out‚3745.08935546875 disk.dm-1.in‚0* proc.memory.percent‚17.177748680114746 proc.cpu.threads‚17 disk.sda2.in‚0 disk.nvme0n1.in‚8.34375 network.sent‚ 2.187723e+06 network.recv‚468946 disk.sda.in‚0 disk.sda.out‚ 53.90234375 disk.drbd2080.out‚0 disk.dm-1.out‚ 53.90234375 disk.drbd2265.out‚0$ memory_percent‚94.16606426239014 disk.sda3.out‚ 53.90234375 disk.sda2.out‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2814.40234375‘B’ò ‚(:‡ ¤ÒãÔÝž gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÇŠFó ‚(:» ¤ÒãÔ¦Ÿø * disk./.usagePercent‚15.912242757485414 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚3745.30419921875$ disk./.usageGB‚237.1039161682129 disk.sda.out‚ 54.30859375 disk.sda2.out‚0 cpu‚25.31603268546998 disk.sda3.out‚ 54.30859375 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2814.40234375' disk.nvme0c0n1.out‚3745.30419921875 disk.dm-0.in‚8.34375 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 5.9052734375* proc.memory.percent‚17.177748680114746 network.sent‚ 2.295249e+06 disk.sda2.in‚0 disk.nvme0c0n1.in‚8.34375 disk.nvme0n1.in‚8.34375 proc.cpu.threads‚17 network.recv‚474816" disk.dm-0.out‚3745.30419921875 disk.drbd2265.out‚0 disk.dm-1.out‚ 54.30859375 disk.drbd2265.in‚0 disk.drbd2266.out‚0# memory_percent‚99.9537467956543$ proc.memory.availableMB‚7.578125‘Î¥12ô - _wandbruntime‚1163  _runtime‚1163ôŸ*Éõ à trainer/global_step‚1499 % _timestamp‚1.7884060607975912e+09 _runtime‚1163.785040932 _step‚119 ' lr-AdamW/pg1‚0.00027457724589719033 ' lr-AdamW/pg2‚0.00027457724589719033ê¤\Íö Ç ' lr-AdamW/pg2‚0.00027457724589719033 trainer/global_step‚1499 % _timestamp‚1.7884060607975912e+09 _runtime‚1163.785040932 _step‚119 ' lr-AdamW/pg1‚0.00027457724589719033wå¹<{2÷ - _wandbruntime‚1163  _runtime‚1163m•YÔÇ'ø ÁO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85843790143951 _runtime‚1163.785995132 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88119087801856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21712376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86866744569344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1315949722624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213775 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85843118666843 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84595403115520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49970356 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28257980 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96198858800640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28257980 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96198858800640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88119087801856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21712376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86866744569344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inacti`»y4vve_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1315949722624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213775 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85843118666843 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84595403115520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49970356 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86923097608192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86923115611136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85899510709683 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96261561391616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1249203463603 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21726390 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21726392 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84650290206720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19894051 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28276460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253162660352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94944559815680 _step‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88176934646272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88176934646272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88176260268544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7218431 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96261380131328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28256398 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84650307246080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28256398 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7218868 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚85898839232367 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12675620 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50002852 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0í íÜÕ&ü ÏM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85899510709683 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96261561391616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1249203463603 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21726390 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21726392 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84650290206720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19894051 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28276460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253162660352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94944559815680 _step‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88176934646272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88176934646272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88176260268544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7218431 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96261380131328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28256398 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84650307246080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28256398 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7218868 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚85898839232367 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12675620 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50002852 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1499 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1253819035136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49982788 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86923115611136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1248549025647 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86923097608192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86923115611136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784y¬¿:`2ý - _wandbruntime‚1164  _runtime‚1164ã (zUþ Ï 2loss/SRqqt1/assignment_loss‚2.5731810098741112 1loss/SRqqt2/assignment_loss‚1.913338379490943 2loss/SRqqt3/assignment_loss‚1.4189530461462159 /loss/FBt1/detection_loss‚3.3088632027308145 /loss/FRt3/assignment_loss‚2.859926425574119 0loss/FRt4/assignment_loss‚4.2375270954587245 0loss/FBt1/assignment_loss‚0.6731291910012563 _runtime‚1164.474365604 /loss/FRt2/assignment_loss‚4.322342365980148 /loss/FRt4/detection_loss‚2.4501867429776625 trainer/global_step‚1499 .loss/FRt2/detection_loss‚2.428733124335607 /loss/FBt3/detection_loss‚0.8650810556252356 .loss/FBt4/detection_loss‚5.440231720606486 % _timestamp‚1.7884060614870148e+09 0loss/FBt4/assignment_loss‚0.3272180110216141 1loss/SRqqt1/detection_loss‚2.2408796817791172 1loss/SRqqt3/detection_loss‚1.3541528839956631 1loss/SRqqt4/assignment_loss‚2.294522378713854 /loss/FRt1/detection_loss‚2.5663575530052185 /loss/SRqqt2/detection_loss‚3.08870054710479 1loss/SRqqt4/detection_loss‚1.9250707468919215 .loss/FRt3/detection_loss‚1.962299522077828 /loss/FBt2/detection_loss‚2.7050091080805836 epoch‚1 /loss/FBt2/assignment_loss‚0.592674571527716 0loss/FBt3/assignment_loss‚0.3660210851731092 %loss/total_loss‚56.27383521941945 _step‚122 0loss/FRt1/assignment_loss‚4.3594357702467175aºZÎYÿ Ó %loss/total_loss‚56.27383521941945 _step‚122 0loss/FRt1/assignment_loss‚4.3594357702467175 2loss/SRqqt1/assignment_loss‚2.5731810098741112 1loss/SRqqt2/assignment_loss‚1.913338379490943 2loss/SRqqt3/assignment_loss‚1.4189530461462159 /loss/FBt1/detection_loss‚3.3088632027308145 /loss/FRt3/assignment_loss‚2.859926425574119 0loss/FRt4/assignment_loss‚4.2375270954587245 0loss/FBt1/assignment_loss‚0.6731291910012563 _runtime‚1164.474365604 /loss/FRt2/assignment_loss‚4.322342365980148 /loss/FRt4/detection_loss‚2.4501867429776625 trainer/global_step‚1499 .loss/FRt2/detection_loss‚2.428733124335607 /loss/FBt3/detection_loss‚0.8650810556252356 .loss/FBt4/detection_loss‚5.440231720606486 % _timestamp‚1.7884060614870148e+09 0loss/FBt4/assignment_loss‚0.3272180110216141 1loss/SRqqt1/detection_loss‚2.2408796817791172 1loss/SRqqt3/detection_loss‚1.3541528839956631 1loss/SRqqt4/assignment_loss‚2.294522378713854 /loss/FRt1/detection_loss‚2.5663575530052185 /loss/SRqqt2/detection_loss‚3.08870054710479 1loss/SRqqt4/detection_loss‚1.9250707468919215 .loss/FRt3/detection_loss‚1.962299522077828 /loss/FBt2/detection_loss‚2.7050091080805836 epoch‚1 /loss/FBt2/assignment_loss‚0.592674571527716 0loss/FBt3/assignment_loss‚0.3660210851731092zÉI$»’€‚(:‡ ³ÒãÔˆþ® gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.777' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7554 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ [c4‚(:© ³ÒãÔ¦…  network.sent‚ 2.301214e+06* disk./.usagePercent‚15.912247621628564 disk.sda1.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2814.40234375% disk./.usageGB‚237.10398864746094 disk.nvme0n1.in‚8.34375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 5.9990234375! disk.nvme0n1.out‚ 3745.7109375 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 network.recv‚479095 disk.sda2.in‚0# disk.nvme0c0n1.out‚ 3745.7109375 disk.dm-0.in‚8.34375 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.sda.out‚54.90625 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 3745.7109375 disk.sda.in‚0 disk.sda3.out‚54.90625* proc.memory.percent‚17.177748680114746 cpu‚25.312689163144434 disk.drbd1716.out‚0 disk.dm-1.out‚54.90625 disk.drbd2266.out‚0 disk.drbd2080.out‚0$ memory_percent‚99.95813369750977$ proc.memory.availableMB‚6.859375 proc.cpu.threads‚17 disk.sda1.in‚0[N0v’‚‚(:‡ ÂÒãÔèÖý gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.136' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8272 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùË«4ƒ‚(:© ÂÒãÔÕŽœ  network.recv‚485301 disk.nvme0c0n1.in‚8.34375" disk.nvme0c0n1.out‚ 3745.921875 disk.nvme0n1.out‚ 3745.921875 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2265.out‚0) disk./.usagePercent‚15.91225274177925 disk.rbd1.in‚0 disk.rbd1.out‚ 6.1708984375 disk.dm-0.in‚8.34375 disk.dm-1.out‚ 55.3515625 disk.drbd2080.out‚0* proc.memory.percent‚17.177772521972656 network.sent‚ 2.409309e+06 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 2814.40625 disk.sda2.in‚0 disk.nvme0n1.in‚8.34375 disk.dm-0.out‚ 3745.921875 disk.sda3.out‚ 55.3515625 proc.cpu.threads‚17% disk./.usageGB‚237.10406494140625 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 cpu‚25.309424845494487 disk.drbd1716.in‚0 disk.sda.out‚ 55.3515625$ memory_percent‚99.95665550231934% proc.memory.availableMB‚ 7.1015625bÆæÚ„‚(:‚ ÑÒãԦᛠgpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚP^…‚(:º ÑÒãÔˆ¿í  disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 6.1708984375& proc.memory.availableMB‚ 7.08984375* disk./.usagePercent‚15.912248901666237 disk.dm-0.in‚8.34375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0! disk.dm-0.out‚3746.9169921875 disk.drbd1716.out‚0$ memory_percent‚99.95672702789307 network.sent‚ 2.415653e+06* proc.memory.percent‚17.177772521972656$ disk.nvme0n1.out‚3746.9169921875 disk.sda.out‚ 55.87109375 disk.sda3.in‚0 disk.sda3.out‚ 55.87109375 disk.sda2.in‚0 disk.nvme0n1.in‚8.34375 disk.drbd2266.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2814.40625 network.recv‚489879% disk./.usageGB‚237.10400772094727 disk.dm-1.out‚ 55.87109375 disk.drbd2265.out‚0 disk.sda.in‚0 disk^ÄF»ç.drbd2080.out‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚3746.9169921875 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd1716.in‚0 cpu‚25.306102574786483 proc.cpu.threads‚17J’b2†- _wandbruntime‚1200  _runtime‚1200kƒ§É‡Ã ' lr-AdamW/pg1‚0.00027439692918517926 ' lr-AdamW/pg2‚0.00027439692918517926 trainer/global_step‚1549 % _timestamp‚1.7884060971614919e+09 _runtime‚1200.148898666 _step‚123zQCÍˆÇ _step‚123 ' lr-AdamW/pg1‚0.00027439692918517926 ' lr-AdamW/pg2‚0.00027439692918517926 trainer/global_step‚1549 % _timestamp‚1.7884060971614919e+09 _runtime‚1200.148898666{¯»Ù2‰- _wandbruntime‚1200  _runtime‚1200¯*hÌÇ'ŠÁO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20518973 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87340609643520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29163996 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1358922105344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1289222367839 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90977725550080 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51597208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22413176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89684414514688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7442699 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22413176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22413174 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90978399914496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚420 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99324368066560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97965469901312 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29184032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29184032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7443119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51577170 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90978399914496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90977725550080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89684396511744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1293985399808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99324547243008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1359077341696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1200.149878384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1293329038336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29163996 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88629832011359 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88629160534251 trainer/global_step‚1549 % _timestamp‚1.7884060971621892e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13075852 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97965445961216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87340592604160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13075854 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1293329038336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22413174 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1293985399808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89684414514688 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89684414514688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7442699 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22413176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22413174 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90978399914496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚420 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99324368066560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97965469901312 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29184032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29184032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7443119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51577170 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90978399914496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90977725550080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89684396511744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1293985399808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99324547243008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1359077341696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1200.149878384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1293329038336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29163996 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88629832011359 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88629160534251 trainer/global_step‚1549 % _timestamp‚1.7884060971621892e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13075852 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97965445961216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87340592604160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13075854 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1293329038336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22413174 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1293985399808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89684414514688 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22427192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89740749550592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98027120638976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98027096698880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1359940977664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88685552577091 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51629704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51609640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7447647 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87395496734720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91035572394496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13084022 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29182450 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91034898016768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7447206 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚1200.839775449 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29182450 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1294804840960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89740767553536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22427190 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87395513774080 % _timestamp‚1.7884060978518822e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89740749550592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20531669 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99387061616640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91035572394496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1359762685952 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29202512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1294148466176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88684881099775 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1290038803011 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1289384365055 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99386859384832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20531226 trainer/global_step‚1549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89740767553536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1294804840960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13084020 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91035572394496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13084022 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29182450 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91034898016768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74­D¾‘ 47206 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚1200.839775449 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29182450 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1294804840960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89740767553536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22427190 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87395513774080 % _timestamp‚1.7884060978518822e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89740749550592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20531669 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99387061616640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91035572394496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1359762685952 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29202512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1294148466176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88684881099775 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1290038803011 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1289384365055 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99386859384832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20531226 trainer/global_step‚1549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89740767553536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1294804840960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13084020 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22427192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89740749550592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98027120638976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98027096698880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1359940977664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88685552577091 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51629704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51609640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7447647 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87395496734720}GU-L2- _wandbruntime‚1200  _runtime‚1200Ì8È,C½ 2loss/SRqqt2/assignment_loss‚2.2817702889442444 .loss/FRt4/detection_loss‚3.145484749476115 1loss/SRqqt1/detection_loss‚2.3852036549494815 1loss/SRqqt3/detection_loss‚1.3912250185385346 1loss/SRqqt4/detection_loss‚1.7775463706861108 -loss/FBt1/detection_loss‚5.22182320356369 trainer/global_step‚1549 /loss/FRt1/assignment_loss‚4.600352103893574 0loss/FBt2/assignment_loss‚0.6584608554840088 /loss/FBt3/detection_loss‚0.7487057458719144 /loss/FRt2/assignment_loss‚4.082020819187164 0loss/FBt1/assignment_loss‚0.4806305030477233 &loss/total_loss‚54.161879515034585 epoch‚1 _step‚126 2loss/SRqqt4/assignment_loss‚1.9312302929870153 0loss/FBt3/assignment_loss‚0.2204178957823529 /loss/FRt3/detection_loss‚1.8942595922698577 /loss/FRt2/detection_loss‚2.7394064217805862 /loss/FBt4/detection_loss‚2.7823386192321777 $ _timestamp‚1.788406097853885e+09 .loss/FRt1/detection_loss‚2.617375827752627 .loss/FRt4/assignment_loss‚4.24054287870725 2loss/SRqqt3/assignment_loss‚1.5473339660093188 0loss/SRqqt2/detection_loss‚2.274268687599235 /loss/FBt2/detection_loss‚2.5364411075909934 1loss/SRqqt1/assignment_loss‚2.209266889935885 loss/FBt4/assignment_loss‚0 _runtime‚1200.841244831 /loss/FRt3/assignment_loss‚2.395774021744728¦ü« G‘Á 1loss/SRqqt3/detection_loss‚1.3912250185385346 1loss/SRqqt4/detection_loss‚1.7775463706861108 -loss/FBt1/detection_loss‚5.22182320356369 trainer/global_step‚1549 /loss/FRt1/assignment_loss‚4.600352103893574 0loss/FBt2/assignment_loss‚0.6584608554840088 /loss/FBt3/detection_loss‚0.7487057458719144 /loss/FRt2/assignment_loss‚4.082020819187164 0loss/FBt1/assignment_loss‚0.4806305030477233 &loss/total_loss‚54.161879515034585 epoch‚1 _step‚126 2loss/SRqqt4/assignment_loss‚1.9312302929870153 0loss/FBt3/assignment_loss‚0.2204178957823529 /loss/FRt3/detection_loss‚1.8942595922698577 /loss/FRt2/detection_loss‚2.7394064217805862 /loss/FBt4/detection_loss‚2.7823386192321777 $ _timestamp‚1.788406097853885e+09 .loss/FRt1/detection_loss‚2.617375827752627 .loss/FRt4/assignment_loss‚4.24054287870725 2loss/SRqqt3/assignment_loss‚1.5473339660093188 0loss/SRqqt2/detection_loss‚2.274268687599235 /loss/FBt2/detection_loss‚2.5364411075909934 1loss/SRqqt1/assignment_loss‚2.209266889935885 loss/FBt4/assignment_loss‚0 _runtime‚1200.841244831 /loss/FRt3/assignment_loss‚2.395774021744728 2loss/SRqqt2/assignment_loss‚2.2817702889442444 .loss/FRt4/detection_loss‚3.145484749476115 1loss/SRqqt1/detection_loss‚2.3852036549494815~ˆ5{’‚(:’ àÒãÔŸ€ž gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.333600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/áÔ E“‚(:º àÒãÔÄœú  disk.dm-1.in‚0 cpu‚25.303000357584196 proc.cpu.threads‚17 network.sent‚ 2.420916e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚8.34375 disk.nvme0n1.in‚8.34375! disk.dm-0.out‚3747.1982421875 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚8.34375 disk.sda.in‚0 disk.sda3.out‚ 56.29296875 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 56.29296875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚3747.1982421875 disk.rbd1.out‚ 6.2333984375 disk.drbd2265.out‚0 disk.sda.out‚ 56.29296875% disk./.usageGB‚237.10400772094727 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0* disk./.usagePercent‚15.912248901666237 disk.drbd2266.in‚0 disk.drbd1716.in‚0$ disk.nvme0n1.out‚3747.1982421875$ memory_percent‚99.95572566986084& proc.memory.availableMB‚ 7.25390625* proc.memory.percent‚17.177772521972656 network.recv‚493818 disk.drbd2080.out‚0 proc.memory.rssMB‚ 2814.40625?µí”‚(:’ ïÒãÔ µÐ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚148.601' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.720200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t)ˆ¾@•‚(:µ ïÒãÔÀïì  disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 56.83203125 disk.sda.in‚0% disk./.usageGB‚237.10305404663086 disk.drbd2080.out‚0 network.recv‚499979 disk.drbd1716.in‚0 disk.rbd1.out‚ 6.2333984375 disk.dm-1.in‚0 disk.sda3.out‚ 56.83203125 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚8.34375# memory_percent‚99.9582290649414$ disk.nvme0n1.out‚3747.5419921875 disk.dm-0.in‚8.34375 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.availableMB‚6.84375* proc.memory.percent‚17.177772521972656! disk.dm-0.out‚3747.5419921875 disk.sda2.out‚0 network.sent‚ 2.529032e+06 disk.drbd1716.out‚0 disk.sda.out‚ 56.83203125 disk.rbd0.in‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚3747.5419921875 disk.nvme0n1.in‚8.34375 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2814.40625 cpu‚25.29978186536633 proc.cpu.threads‚17* disk./.usagePercent‚15.912184899782677÷"^Ç2–- _wandbruntime‚1236  _runtime‚1236ôxÇ—Á _runtime‚1236.933700456 _step‚127 & lr-AdamW/pg1‚0.0002741880895112586 & lr-AdamW/pg2‚0.0002741880895112586 trainer/global_step‚1599 % _timestamp‚1.7884061339462705e+09ÎO<¤Ë˜Å _runtime‚1236.933700456 _step‚127 & lr-AdamW/pg1‚0.0002741880895112586 & lr-AdamW/pg2‚0.0002741880895112586 trainer/global_step‚1599 % _timestamp‚1.7884061339462705e+09ŽÓ|2™- _wandbruntime‚1236  _runtime‚1236’|ˆ/Ë'šÅO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23113976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30090048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102449824094720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92502066457088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91415202401659 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92502048454144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90085816171520 % _timestamp‚1.7884061339470246e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92502066457088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93836363298304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53224060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91415873878767 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53204022 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30110084 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13484252 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92502048454144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7671449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23113976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1236.934634899 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21155701 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93837037662720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1334314844160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1402002778112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90085799132160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1330057707247 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93836363298304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1334314844160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1334971205632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93837037662720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23113974 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21156154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23113974 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30110084 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13484252 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92502048454144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7671449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23113976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1236.934634899 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21155701 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93837037662720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1334314844160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1402002778112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90085799132160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1330057707247 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93836363298304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1334314844160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1334971205632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93837037662720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23113974 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21156154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23113974 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92502066457088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91415202401659 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92502048454144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90085816171520 % _timestamp‚1.7884061339470246e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92502066457088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93836363298304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53224060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91415873878767€^±2œ- _wandbruntime‚1237  _runtime‚12374“ÉñÕ&ÏM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102512349182464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30128564 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23127990 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53236492 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21168370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492420 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚129 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53256556 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93894210142720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1402872920576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1330219704463 % _timestamp‚1.7884061346276443e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1335790646784 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92558419495936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492422 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1335134272000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23127990 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92558401492992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23127992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90140720302080 trainer/global_step‚1599 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93893535764992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23127992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30108502 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7675950 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¨‹N[Ú&žÔM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30108502 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7675950 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102512349182464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30128564 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23127990 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53236492 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21168370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492420 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚129 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53256556 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93894210142720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1402872920576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1330219704463 % _timestamp‚1.7884061346276443e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1335790646784 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92558419495936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492422 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1335134272000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23127990 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92558401492992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23127992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90140720302080 trainer/global_step‚1599 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93893535764992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23127992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105575462412800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95319700396544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95319718399488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21791134 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7898017 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104130543667712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370893046655 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1444942685184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31016100 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375957011456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13892654 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92831022699520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23814774 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96695001046528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1375300649984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95319700396544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54830874 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96695675410944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94201915746175 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94201244269067 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31036136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23814776 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105575662560768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚1273.278737577°+*¹Ð'«ÊO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95319700396544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95319718399488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21791134 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7898017 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104130543667712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370893046655 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1444942685184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31016100 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375957011456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13892654 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92831022699520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23814774 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96695001046528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1375300649984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95319700396544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54830874 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96695675410944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94201915746175 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94201244269067 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31036136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23814776 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105575662560768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚1273.278737577 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95319718399488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1375957011456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96695001046528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105575462412800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0„g¼ü2¬- _wandbruntime‚1273  _runtime‚1273nÈÖÕ&­ÏM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23828792 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7902481 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1445964343808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚1649 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96752847890944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92885926830080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96752847890944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7902932 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚1273.966342539 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96752173513216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884061709784393e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1445792343552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31054616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95376053435392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31054616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95376071438336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54863344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371709481827 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23828790 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13900820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104192194405376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23828792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54863344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21803301 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94257636311907 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95376071438336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13900822 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1371055043871 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚133 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96752847890944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7902932 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚1273.966342539 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96752173513216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884061709784393e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1445792343552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31054616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95376053435392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31054616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95376071438336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54863344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371709481827 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23828790 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13900820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104192194405376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocatiof_KN ns.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23828792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54863344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21803301 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94257636311907 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95376071438336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13900822 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1371055043871 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚133 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F³‚(:³ œÓãÔš¸Û  proc.cpu.threads‚17" disk.dm-0.out‚3748.66943359375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 58.453125 disk.drbd1716.out‚0$ memory_percent‚99.95644092559814 disk.sda1.out‚0 disk.rbd1.out‚ 6.5234375 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2814.40625* proc.memory.percent‚17.177772521972656 network.recv‚513837$ disk./.usageGB‚237.1032943725586 disk.sda3.in‚0 disk.nvme0c0n1.in‚8.34375' disk.nvme0c0n1.out‚3748.66943359375 disk.drbd1716.in‚0 disk.dm-0.in‚8.34375 disk.sda.in‚0 disk.sda2.out‚0* disk./.usagePercent‚15.912201028257334 disk.drbd2080.in‚0 network.sent‚ 2.647898e+06 disk.sda3.out‚ 58.453125 disk.rbd0.out‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 7.13671875 cpu‚25.291060317754376 disk.sda2.in‚0 disk.nvme0n1.in‚8.34375% disk.nvme0n1.out‚3748.66943359375 disk.dm-1.in‚0 disk.dm-1.out‚ 58.453125ùñ’´‚(:‡ «ÓãÔÔã› gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ütR4µ‚(:© «ÓãÔŽ—ƒ % proc.memory.availableMB‚ 7.1328125 cpu‚25.28833854429114$ disk.nvme0c0n1.out‚ 3749.95703125 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0$ memory_percent‚99.95646476745605 network.sent‚ 2.756237e+06 disk.drbd1716.out‚0" disk.nvme0n1.out‚ 3749.95703125 disk.drbd2265.out‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 2814.40625 proc.cpu.threads‚17 disk.rbd0.out‚0 network.recv‚520057% disk./.usageGB‚237.10329818725586 disk.rbd1.out‚ 6.5234375 disk.dm-0.in‚8.34375 disk.sda.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.177772521972656 disk.nvme0n1.in‚8.34375 disk.sda.out‚ 59.109375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚8.34375 disk.sda3.out‚ 59.109375* disk./.usagePercent‚15.912201284264867 disk.dm-0.out‚ 3749.95703125 disk.dm-1.out‚ 59.109375g;VV¶‚(:’ ºÓãÔ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚131.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.262400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òXãÒ8·‚(:­ ºÓãÔ†Èé  disk.rbd0.in‚0 disk.drbd2266.in‚0" disk.nvme0n1.out‚ 3750.16015625 disk.drbd2265.in‚0 disk.dm-0.in‚8.34375* disk./.usagePercent‚15.912205124377882 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd1716.in‚0 disk.sda.in‚0 network.sent‚ 2.762063e+06 network.recv‚524538 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375 disk.sda3.in‚0 disk.sda3.out‚ 59.578125 cpu‚25.28532179602002$ disk.nvme0c0n1.out‚ 3750.16015625 disk.dm-0.out‚ 3750.16015625 disk.dm-1.out‚ 59.578125* proc.memory.percent‚17.177772521972656 disk.dm-1.in‚0 disk.sda.out‚ 59.578125 proc.memory.rssMB‚ 2814.40625 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd1.out‚ 6.6650390625$ memory_percent‚99.95839595794678% disk./.usageGB‚237.10335540771484 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0& proc.memory.availableMB‚ 6.81640625 disk.drbd2266.out‚0RJÝ´2¸- _wandbruntime‚1310  _runtime‚1310]¯UƹÀ & lr-AdamW/pg2‚0.0002736850213002615 trainer/global_step‚1699 $ _timestamp‚1.788406207095033e+09 _runtime‚1310.082495199 _step‚135 & lr-AdamW/pg1‚0.0002736850213002615÷ÊÄ˺Š& lr-AdamW/pg1‚0.0002736850213002615 & lr-AdamW/pg2‚0.0002736850213002615 trainer/global_step‚1699 $ _timestamp‚1.788406207095033e+09 _runtime‚1310.082495199 _step‚135‡&=Dþ2»- _wandbruntime‚1310  _runtime‚1310ã£TDË'¼ÅO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98137370341888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99554313159168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416942817280 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99554313159168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487939000320 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99553638794752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8125941 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31962188 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108700995611136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14301054 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98137352338944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1411728386063 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31942152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108701195759104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107213080550912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99553638794752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98137370341888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95576212188160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1416942817280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1488115208192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1411073948315 _step‚136ÆX0Ð'½ÊO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99554313159168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416942817280 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99554313159168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487939000320 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99553638794752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8125941 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31962188 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108700995611136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14301054 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98137352338944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1411728386063 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31942152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108701195759104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107213080550912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99553638794752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98137370341888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95576212188160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1416942817280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1488115208192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1411073948315 _step‚136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98137370341888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872ˆº¤ï!2¾- _wandbruntime‚1310  _runtime‚1310•ƒ¿ÏM R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97043006701999 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108763693703680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1488794609152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95631133358080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98193705377792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batcný‰Ö¶"h_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1488962415104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1310.777497243 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14309222 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107274707348480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884062077895727e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14309220 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1417105883648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24529592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚1699 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24529592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99610811261440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95631116318720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108763501957632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22439229 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417762258432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1411890383279 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56490196 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24529590 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31980668 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99611485639168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56490196 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97043678179315 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24529590 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31980668 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98193705377792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1417105883648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8130452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412544821235 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98193723380736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107274731288576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99611485639168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56510260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31960606 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 _step‚137 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98193723380736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844@ìñÚ&ÀÔM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24529590 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31980668 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99611485639168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56490196 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97043678179315 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24529590 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31980668 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98193705377792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1417105883648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8130452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412544821235 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98193723380736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107274731288576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99611485639168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56510260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31960606 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 _step‚137 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98193723380736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97043006701999 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108763693703680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1488794609152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95631133358080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98193705377792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1488962415104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1310.777497243 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14309222 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107274707348480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884062077895727e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14309220 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1417105883648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24529592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚1699 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24529592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99610811261440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95631116318720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108763501957632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22439229 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417762258432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1411890383279 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56490196 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928‰®¿ç2Á- _wandbruntime‚1310  _runtime‚1310ãªY|XÂÒ 0loss/FBt1/assignment_loss‚0.4641595108339166 0loss/FBt2/assignment_loss‚0.6268217414617367 1loss/SRqqt4/detection_loss‚1.9089775521840369 -loss/FBt2/detection_loss‚1.90413998067379 2loss/SRqqt2/assignment_loss‚2.1573753245174885 1loss/SRqqt3/assignment_loss‚1.402697989434907 2loss/SRqqt4/assignment_loss‚1.9024933094957046 .loss/FRt4/detection_loss‚2.567731241385142 .loss/FBt4/detection_loss‚4.134675184885661 _runtime‚1310.778778039 _step‚138 1loss/SRqqt1/assignment_loss‚2.047082995826548 1loss/SRqqt2/detection_loss‚2.7967204501231513 0loss/FRt2/assignment_loss‚4.5301367849917025 0loss/FRt3/assignment_loss‚2.8798516773163003 /loss/FRt4/assignment_loss‚3.834061012381599 1loss/FBt4/assignment_loss‚0.10788379516452551 /loss/FRt3/detection_loss‚2.3060778672707842 0loss/FRt1/assignment_loss‚3.9642791197850156 1loss/FBt3/assignment_loss‚0.23618672299381743 /loss/FRt1/detection_loss‚2.6190119553834963 %loss/total_loss‚52.37511409720963 trainer/global_step‚1699 0loss/SRqqt1/detection_loss‚2.534625575397954 0loss/SRqqt3/detection_loss‚1.231301010924984 % _timestamp‚1.7884062077914026e+09 /loss/FRt2/detection_loss‚2.5234308951609843 /loss/FBt1/detection_loss‚2.7977963181642385 /loss/FBt3/detection_loss‚0.8975960814521468 epoch‚1‡þ™¨]Ã× /loss/FRt2/detection_loss‚2.5234308951609843 /loss/FBt1/detection_loss‚2.7977963181642385 /loss/FBt3/detection_loss‚0.8975960814521468 epoch‚1 0loss/FBt1/assignment_loss‚0.4641595108339166 0loss/FBt2/assignment_loss‚0.6268217414617367 1loss/SRqqt4/detection_loss‚1.9089775521840369 -loss/FBt2/detection_loss‚1.90413998067379 2loss/SRqqt2/assignment_loss‚2.1573753245174885 1loss/SRqqt3/assignment_loss‚1.402697989434907 2loss/SRqqt4/assignment_loss‚1.9024933094957046 .loss/FRt4/detection_loss‚2.567731241385142 .loss/FBt4/detection_loss‚4.134675184885661 _runtime‚1310.778778039 _step‚138 1loss/SRqqt1/assignment_loss‚2.047082995826548 1loss/SRqqt2/detection_loss‚2.7967204501231513 0loss/FRt2/assignment_loss‚4.5301367849917025 0loss/FRt3/assignment_loss‚2.8798516773163003 /loss/FRt4/assignment_loss‚3.834061012381599 1loss/FBt4/assignment_loss‚0.10788379516452551 /loss/FRt3/detection_loss‚2.3060778672707842 0loss/FRt1/assignment_loss‚3.9642791197850156 1loss/FBt3/assignment_loss‚0.23618672299381743 /loss/FRt1/detection_loss‚2.6190119553834963 %loss/total_loss‚52.37511409720963 trainer/global_step‚1699 0loss/SRqqt1/detection_loss‚2.534625575397954 0loss/SRqqt3/detection_loss‚1.231301010924984 % _timestamp‚1.7884062077914026e+09ŠÒ¯ƒKÄ‚(:… ÉÓãÔŠÔŽ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØþDÅ‚(:¹ ÉÓãÔîø‚  disk.rbd0.out‚0 disk.sda1.in‚0 network.sent‚ 2.768302e+06% disk./.usageGB‚237.10329818725586' disk.nvme0c0n1.out‚3750.59521484375 disk.dm-0.in‚8.34375 disk.drbd2265.in‚0 disk.sda.out‚ 60.20703125 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 6.6650390625" disk.dm-0.out‚3750.59521484375 disk.drbd2265.out‚0 disk.sda.in‚0 proc.memory.rssMB‚ 2814.40625 disk.sda3.out‚ 60.20703125% disk.nvme0n1.out‚3750.59521484375 disk.dm-1.out‚ 60.20703125 network.recv‚529204 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0# proc.memory.availableMB‚6.53125 cpu‚25.28250379392856 proc.cpu.threads‚17 disk.dm-1.in‚0* disk./.usagePercent‚15.912201284264867 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚8.34375$ memory_percent‚99.96013641357422* proc.memory.percent‚17.177772521972656 disk.sda1.out‚0ëxëÊÆ‚(:… ØÓãÔÔÖó gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓFqHEÇ‚(:º ØÓãÔòž—  disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 60.6953125 disk.drbd2266.in‚0 disk.nvme0n1.in‚8.34375$ memory_percent‚99.95949268341064& proc.memory.availableMB‚ 6.63671875* disk./.usagePercent‚15.912201284264867% disk./.usageGB‚237.10329818725586 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 6.7275390625* proc.memory.percent‚17.177772521972656 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda.out‚ 60.6953125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚3750.81396484375 disk.dm-0.in‚8.34375 disk.sda3.out‚ 60.6953125 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375 cpu‚25.279894851894436 network.sent‚ 2.875772e+06 network.recv‚534364" disk.dm-0.out‚3750.81396484375 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚3750.81396484375 proc.memory.rssMB‚ 2814.40625 disk.drbd1716.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0#T2È- _wandbruntime‚1346  _runtime‚1346ÿT´YÉÉà _runtime‚1346.346167701 _step‚139 ' lr-AdamW/pg1‚0.00027339089760184624 ' lr-AdamW/pg2‚0.00027339089760184624 trainer/global_step‚1749 % _timestamp‚1.7884062433586993e+09<ƒsÎÊÈ _step‚139 ' lr-AdamW/pg1‚0.00027339089760184624 ' lr-AdamW/pg2‚0.00027339089760184624 trainer/global_step‚1749 % _timestamp‚1.7884062433586993e+09 _runtime‚1346.346167701‹Wo2Ë- _wandbruntime‚1346  _runtime‚1346 íäåƒÌÌO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1451909287723 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚102412950907392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25216374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102412276542976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14709452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98321435755520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110295593494016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25216376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32868204 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98321418716160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100955022284288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1531238647808 _step‚140 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1457272261632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58084578 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1457928623104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32888240 M@DeviceStatsMÝÎÓO onitor.on_train_batch_start/inactive_split.all.freed‚23062576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102412950907392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25216374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32888240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1457272261632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23063008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100955022284288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32868204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102412276542976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1749 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25216376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32868204 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98321418716160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100955022284288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1531238647808 _step‚140 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1457272261632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58084578 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1457928623104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32888240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23062576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102412950907392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25216374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32888240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1457272261632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23063008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100955022284288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32868204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102412276542976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1749 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14709452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98321435755520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110295593494016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23075201 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1531922010624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚141 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32906720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110357244231680 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1458091689472 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1458748064256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58117048 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58117048 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32886658 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102470123387392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111889166242304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1453380160643 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884062440342057e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32886658 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102469449009664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25230390 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101011357320192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1532100302336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14717622 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1458091689472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111889368474112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102470123387392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98376339886080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25230392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Sü¨&ÐÜM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102469449009664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25230390 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101011357320192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1532100302336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14717622 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1458091689472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111889368474112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102470123387392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98376339886080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25230392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1458748064256 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99829048569407 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101011375323136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110357268171776 trainer/global_step‚1749 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25230390 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚99829720046723 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98376322846720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1452725722687 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23075650 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚1347.022205897 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14717620 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8358028 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58137112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32906720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23075201 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1531922010624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚141 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32906720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110357244231680 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1458091689472 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1458748064256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58117048 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58117048 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32886658 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102470123387392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111889166242304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1453380160643 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884062440342057e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.{_ :on_train_batch_end/active.small_pool.freed‚32886658ƒtð2Ñ- _wandbruntime‚1347  _runtime‚1347'Û‡XÒÒ /loss/FRt4/assignment_loss‚4.483289588380743 /loss/FRt4/detection_loss‚2.4111413304452545 1loss/SRqqt4/detection_loss‚1.7646051654175146 .loss/FBt2/detection_loss‚2.795136166115602 /loss/FBt4/detection_loss‚2.6896611213684083 0loss/FRt3/assignment_loss‚3.0069209751875503 0loss/FBt4/assignment_loss‚0.5758671671152115 % _timestamp‚1.7884062440359714e+09 1loss/SRqqt1/assignment_loss‚2.240360673178326 2loss/SRqqt4/assignment_loss‚2.0934709635243487 /loss/FRt1/detection_loss‚2.6580883967570768 %loss/total_loss‚53.35265045145097 _runtime‚1347.023318416 2loss/SRqqt3/assignment_loss‚1.8227741165229907 /loss/FRt2/detection_loss‚2.3935058677897736 1loss/SRqqt2/detection_loss‚2.1151942763573084 /loss/FBt3/detection_loss‚0.8053488399054313 0loss/FRt1/assignment_loss‚3.8688601561081715 /loss/FRt2/assignment_loss‚4.546828529414008 1loss/SRqqt2/assignment_loss‚2.312056900216983 0loss/SRqqt1/detection_loss‚2.165745648470792 0loss/FBt1/assignment_loss‚0.7797637222071595 0loss/FBt3/assignment_loss‚0.2598707538597219 /loss/FRt3/detection_loss‚1.8493006818968316 0loss/SRqqt3/detection_loss‚1.471535905278646 /loss/FBt1/detection_loss‚3.7298271507024765 trainer/global_step‚1749 _step‚142 0loss/FBt2/assignment_loss‚0.5134963552306431 epoch‚1|Vþª]Ó× epoch‚1 /loss/FRt4/assignment_loss‚4.483289588380743 /loss/FRt4/detection_loss‚2.4111413304452545 1loss/SRqqt4/detection_loss‚1.7646051654175146 .loss/FBt2/detection_loss‚2.795136166115602 /loss/FBt4/detection_loss‚2.6896611213684083 0loss/FRt3/assignment_loss‚3.0069209751875503 0loss/FBt4/assignment_loss‚0.5758671671152115 % _timestamp‚1.7884062440359714e+09 1loss/SRqqt1/assignment_loss‚2.240360673178326 2loss/SRqqt4/assignment_loss‚2.0934709635243487 /loss/FRt1/detection_loss‚2.6580883967570768 %loss/total_loss‚53.35265045145097 _runtime‚1347.023318416 2loss/SRqqt3/assignment_loss‚1.8227741165229907 /loss/FRt2/detection_loss‚2.3935058677897736 1loss/SRqqt2/detection_loss‚2.1151942763573084 /loss/FBt3/detection_loss‚0.8053488399054313 0loss/FRt1/assignment_loss‚3.8688601561081715 /loss/FRt2/assignment_loss‚4.546828529414008 1loss/SRqqt2/assignment_loss‚2.312056900216983 0loss/SRqqt1/detection_loss‚2.165745648470792 0loss/FBt1/assignment_loss‚0.7797637222071595 0loss/FBt3/assignment_loss‚0.2598707538597219 /loss/FRt3/detection_loss‚1.8493006818968316 0loss/SRqqt3/detection_loss‚1.471535905278646 /loss/FBt1/detection_loss‚3.7298271507024765 trainer/global_step‚1749 _step‚142 0loss/FBt2/assignment_loss‚0.5134963552306431Ž¥oÎ"’Ô‚(:‡ çÓãÔíþ¦ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö­^£AÕ‚(:¶ çÓãÔô†˜  disk.drbd2080.in‚0# proc.memory.rssMB‚ 2814.41015625* disk./.usagePercent‚15.912201796279938$ disk./.usageGB‚237.1033058166504 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚8.34375 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 61.36328125* proc.memory.percent‚17.177796363830566 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 3751.51171875 disk.drbd1716.in‚0 disk.dm-1.out‚ 61.36328125 disk.rbd0.out‚0 network.recv‚538480 disk.rbd1.out‚ 6.7275390625 disk.dm-0.in‚8.34375 disk.sda.out‚ 61.36328125 disk.drbd2266.out‚0 cpu‚25.277061012651828 proc.cpu.threads‚17 disk.sda2.out‚0" disk.nvme0n1.out‚ 3751.51171875 disk.dm-0.out‚ 3751.51171875 disk.sda.in‚0 disk.drbd1716.out‚0$ memory_percent‚99.95858669281006 disk.nvme0n1.in‚8.34375 disk.drbd2080.out‚0& proc.memory.availableMB‚ 6.78515625 network.sent‚ 2.881851e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.in‚02½’Ö‚(:‡ öÓãÔÄ‹° gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.108' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6216 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÈR¾6ׂ(:« öÓãÔî  disk.sda.in‚0 disk.sda.out‚ 62.00390625# proc.memory.rssMB‚ 2814.41015625* proc.memory.percent‚17.177796363830566 disk.sda2.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 network.sent‚ 2.989938e+06 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.out‚ 3751.78125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.dm-0.in‚8.34375$ memory_percent‚99.95718002319336$ proc.memory.availableMB‚7.015625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 62.00390625 disk.drbd2265.in‚0 disk.dm-0.out‚ 3751.78125 disk.sda1.out‚0 network.recv‚544485* disk./.usagePercent‚15.912205636392951% disk./.usageGB‚237.10336303710938 disk.sda3.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.out‚ 3751.78125 disk.sda3.out‚ 62.00390625 disk.nvme0c0n1.in‚8.34375 cpu‚25.274429388353777 disk.drbd2266.in‚0 disk.rbd1.out‚ 6.869140625 disk.nvme0n1.in‚8.34375 disk.dm-1.in‚0šløb‘Ø‚(:† …ÔãÔ•¶ gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ‚P?<Ù‚(:± …ÔãÔÖ‡•  disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 7.341796875 disk.dm-0.in‚8.34375 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2814.41015625 proc.cpu.threads‚17 network.recv‚548956 disk.drbd2080.out‚0& disk.nvme0c0n1.out‚3752.7490234375 disk.sda.out‚ 62.640625 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375* proc.memory.percent‚17.177796363830566 disk.sda1.out‚0 disk.nvme0c0n1.in‚8.34375 disk.dm-1.out‚ 62.640625$ memory_percent‚99.95765686035156" proc.memory.availableMB‚6.9375* disk./.usagePercent‚15.912201796279938 disk.rbd0.out‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚3752.7490234375! disk.dm-0.out‚3752.7490234375 cpu‚25.272035823613628 network.sent‚ 2.996226e+06$ disk./.usageGB‚237.1033058166504 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 62.640625¸ñTÛ2Ú- _wandbruntime‚1382  _runtime‚1382|ûžqÇÛÁ trainer/global_step‚1799 % _timestamp‚1.7884062797378998e+09 _runtime‚1382.725358266 _step‚143 & lr-AdamW/pg1‚0.0002730684605969955 & lr-AdamW/pg2‚0.0002730684605969955‡éÈ}ÌÜÆ & lr-AdamW/pg1‚0.0002730684605969955 & lr-AdamW/pg2‚0.0002730684605969955 trainer/global_step‚1799 % _timestamp‚1.7884062797378998e+09 _runtime‚1382.725358266 _step‚143C0>Á2Ý- _wandbruntime‚1382  _runtime‚1382C·×'ÞÑO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33814292 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8581615 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103772674226688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59731468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114952202439680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33794256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚1799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25917174 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103772656223744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498258067456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105270914291200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23699887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15117854 _runtime‚1382.726232545 _step‚144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101066625244160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1498914428928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105271588655616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114952387907584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884062797386215e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814292 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25917176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1574233590272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105270914291200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25917176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103772674226688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33794256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1574072062464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1498914428928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23699467 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15117852 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113378154317312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1493399064879 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102559369871291©Ó˜Ü'ßÖO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103772656223744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498258067456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105270914291200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23699887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15117854 _runtime‚1382.726232545 _step‚144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101066625244160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1498914428928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105271588655616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114952387907584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884062797386215e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814292 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25917176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1574233590272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105270914291200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25917176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103772674226688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33794256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1574072062464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1498914428928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23699467 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15117852 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113378154317312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1493399064879 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102559369871291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33814292 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8581615 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103772674226688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59731468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114952202439680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33794256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚1799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25917174×­£ß2à- _wandbruntime‚1383  _runtime‚1383ãiU¡áÚM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33832772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33812710 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8586533 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1493561062095 _runtime‚1383.408119814 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33832772 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1499733870080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105328761135616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105328761135616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103829027265536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25931190 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59743900 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33812710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59743900 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59763964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115014703057920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103829009262592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 C9Devi»ÎŸuÃceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15126020 trainer/global_step‚1799 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788406280420162e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101121546414080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105328086757888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1574921943040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25931190 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103829027265536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15126022 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23712555 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102615090436815 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113439805054976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103829009262592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1499733870080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1499077495296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23712111 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360/ˆ±ºå&âßM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788406280420162e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101121546414080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105328086757888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1574921943040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25931190 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103829027265536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15126022 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23712555 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102615090436815 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113439805054976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103829009262592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1499733870080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1499077495296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23712111 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33832772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33812710 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8586533 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1493561062095 _runtime‚1383.408119814 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33832772 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1499733870080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105328761135616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105328761135616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103829027265536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25931190 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59743900 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33812710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59743900 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59763964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115014703057920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103829009262592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15126020 trainer/global_step‚1799 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430‘£Ú:•2ã- _wandbruntime‚1383  _runtime‚1383u#QäË .loss/FBt4/detection_loss‚4.145249366760254 /loss/FBt3/detection_loss‚0.5599143458634936 _step‚146 /loss/FRt4/assignment_loss‚3.854417324066162 2loss/SRqqt4/assignment_loss‚2.0581982389092444 0loss/FBt3/assignment_loss‚0.2903534329560841 .loss/FRt1/detection_loss‚2.896942348315798 1loss/SRqqt4/detection_loss‚1.7857432067394257 # _timestamp‚1.78840628042221e+09 /loss/FRt1/assignment_loss‚4.328953060610541 0loss/FBt2/assignment_loss‚0.6049964262970856 .loss/FRt4/detection_loss‚2.612480120360851 %loss/total_loss‚53.70127158065607 epoch‚1 /loss/FRt2/assignment_loss‚4.927524587592563 1loss/SRqqt1/assignment_loss‚2.055088383230296 trainer/global_step‚1799 1loss/FBt1/assignment_loss‚0.32681877351942035 0loss/SRqqt1/detection_loss‚2.237061405723745 1loss/SRqqt3/detection_loss‚1.3628213748074414 /loss/FBt1/detection_loss‚3.8366278409957886 .loss/FRt2/detection_loss‚2.468450198302398 .loss/FBt2/detection_loss‚2.066887827146621 _runtime‚1383.409559038 0loss/FBt4/assignment_loss‚0.4985125348903239 /loss/FRt3/detection_loss‚1.8875810065203242 /loss/FRt3/assignment_loss‚2.498459843573747 1loss/SRqqt2/assignment_loss‚2.392595141448758 2loss/SRqqt3/assignment_loss‚1.8157701153765644 0loss/SRqqt2/detection_loss‚2.189824676649137HŒ|VåÐ /loss/FRt4/assignment_loss‚3.854417324066162 2loss/SRqqt4/assignment_loss‚2.0581982389092444 0loss/FBt3/assignment_loss‚0.2903534329560841 .loss/FRt1/detection_loss‚2.896942348315798 1loss/SRqqt4/detection_loss‚1.7857432067394257 # _timestamp‚1.78840628042221e+09 /loss/FRt1/assignment_loss‚4.328953060610541 0loss/FBt2/assignment_loss‚0.6049964262970856 .loss/FRt4/detection_loss‚2.612480120360851 %loss/total_loss‚53.70127158065607 epoch‚1 /loss/FRt2/assignment_loss‚4.927524587592563 1loss/SRqqt1/assignment_loss‚2.055088383230296 trainer/global_step‚1799 1loss/FBt1/assignment_loss‚0.32681877351942035 0loss/SRqqt1/detection_loss‚2.237061405723745 1loss/SRqqt3/detection_loss‚1.3628213748074414 /loss/FBt1/detection_loss‚3.8366278409957886 .loss/FRt2/detection_loss‚2.468450198302398 .loss/FBt2/detection_loss‚2.066887827146621 _runtime‚1383.409559038 0loss/FBt4/assignment_loss‚0.4985125348903239 /loss/FRt3/detection_loss‚1.8875810065203242 /loss/FRt3/assignment_loss‚2.498459843573747 1loss/SRqqt2/assignment_loss‚2.392595141448758 2loss/SRqqt3/assignment_loss‚1.8157701153765644 0loss/SRqqt2/detection_loss‚2.189824676649137 .loss/FBt4/detection_loss‚4.145249366760254 /loss/FBt3/detection_loss‚0.5599143458634936 _step‚146’,žTþœæ‚(:‘ ”ÔãÔŸ‘ˆ gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.858' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.77160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒNçn9ç‚(:® ”ÔãÔšÌþ  disk.rbd1.out‚ 7.513671875 disk.nvme0n1.in‚8.34375! disk.dm-0.out‚3753.0458984375 disk.sda3.out‚63.0625 disk.rbd0.in‚0& proc.memory.availableMB‚ 6.90234375 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0$ memory_percent‚99.95787143707275 cpu‚25.269696950572115 network.sent‚ 3.00154e+06* disk./.usagePercent‚15.912201796279938 disk.dm-1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 network.recv‚552661 disk.dm-0.in‚8.34375 disk.drbd2265.out‚0 disk.sda.out‚63.0625 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚63.0625 disk.drbd2266.in‚0 disk.drbd2266.out‚0& disk.nvme0c0n1.out‚3753.0458984375# proc.memory.rssMB‚ 2814.41015625$ disk./.usageGB‚237.1033058166504 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚3753.0458984375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375* proc.memory.percent‚17.177796363830566G’_v’è‚(:‡ £ÔãÔÓ‹§ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j-œ&:é‚(:¯ £ÔãÔ¼±Ó  disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375" disk.dm-0.out‚3753.44384765625 disk.sda.out‚63.625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.drbd2265.in‚0$ disk./.usageGB‚237.1033058166504% disk.nvme0n1.out‚3753.44384765625 disk.dm-1.in‚0$ memory_percent‚99.95791912078857 disk.rbd1.out‚ 7.513671875 disk.sda3.out‚63.625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 cpu‚25.267413107679143 disk.sda2.out‚0 disk.drbd1716.in‚0 network.recv‚558994 disk.sda1.out‚0 disk.nvme0c0n1.in‚8.34375' disk.nvme0c0n1.out‚3753.44384765625 disk.dm-1.out‚63.625 disk.sda3.in‚0& proc.memory.availableMB‚ 6.89453125# proc.memory.rssMB‚ 2814.41015625 disk.drbd2265.out‚0 network.sent‚ 3.109778e+06* disk./.usagePercent‚15.912201796279938 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚8.34375* proc.memory.percent‚17.177796363830566 proc.cpu.threads‚17 disk.sda2.in‚0:ÃY2ê- _wandbruntime‚1418  _runtime‚14184°µÇëÁ & lr-AdamW/pg1‚0.0002727177774810979 & lr-AdamW/pg2‚0.0002727177774810979 trainer/global_step‚1849 % _timestamp‚1.7884063157118418e+09 _runtime‚1418.699280053 _step‚147–û\ÌìÆ _step‚147 & lr-AdamW/pg1‚0.0002727177774810979 & lr-AdamW/pg2‚0.0002727177774810979 trainer/global_step‚1849 % _timestamp‚1.7884063157118418e+09 _runtime‚1418.699280053“Á32í- _wandbruntime‚1418  _runtime‚1418¾Óî×'îÑO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1533579966539 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1539900234752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61338282 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26617976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106590308166144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1617351284736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1534234404287 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1617179271168 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108129552039424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116460691200512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106590326169088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34740344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24336753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108130226403840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8810037 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105346083215807 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116460667260416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108130226403840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15526252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884063157125647e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 trainer/global_step‚1849 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106590308166144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1539243873280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26617976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108129552039424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568¬½. §ïÖO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108129552039424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1533579966539 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1539900234752 N@DeviceStatsMonitor.on_train_batch_start/åöÒÖ5%active_bytes.all.current‚ 674364416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61338282 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26617976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106590308166144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1617351284736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1534234404287 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1617179271168 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108129552039424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116460691200512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106590326169088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34740344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24336753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108130226403840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8810037 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105346083215807 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116460667260416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108130226403840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15526252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884063157125647e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 trainer/global_step‚1849 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106590308166144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1539243873280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26617976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928”î! û2ð- _wandbruntime‚1419  _runtime‚1419óœóá&ñÛM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105401132304223 _runtime‚1419.384887357 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34738762 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26631990 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118140549621248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108187398883840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116522317998080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540063301120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1534396401503 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105401803781539 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚106646679207936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118140362069504 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61390816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34758824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61370752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103866752942080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚106646661204992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540719675904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15534422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8815047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116522341938176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103866735902720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1540719675904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108187398883840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1618207683072 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26631992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26631990 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 trainer/global_step‚1849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108186724506112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884063163969963e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34758824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106646661204992 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61370752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26631992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8814599 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1618044071424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24349019 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1535050839459 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952Èz^ æ&òàM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26631992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26631990 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 trainer/global_step‚1849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108186724506112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884063163969963e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34758824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106646661204992 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61370752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26631992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8814599 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1618044071424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24349019 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1535050839459 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105401132304223 _runtime‚1419.384887357 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34738762 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26631990 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118140549621248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108187398883840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116522317998080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540063301120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1534396401503 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105401803781539 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚106646679207936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118140362069504 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61390816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34758824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61370752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103866752942080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚106646661204992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540719675904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15534422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8815047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116522341938176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103866735902720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1540719675904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108187398883840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1618207683072 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0•šV9±2ó- _wandbruntime‚1419  _runtime‚1419bÑçqUôÏ _step‚150 /loss/FRt2/assignment_loss‚5.336807693753924 1loss/SRqqt1/assignment_loss‚2.465936025886825 /loss/FRt3/detection_loss‚1.8066625259086198 /loss/FBt4/detection_loss‚4.2727073431015015 0loss/FRt3/assignment_loss‚2.6088051199913025 /loss/FBt1/assignment_loss‚0.310475192964077 /loss/FRt4/detection_loss‚2.2391393921285307 %loss/total_loss‚55.77098063953024 epoch‚1 0loss/SRqqt1/detection_loss‚2.548379985672055 0loss/SRqqt2/detection_loss‚2.829245748122533 0loss/SRqqt3/detection_loss‚1.326656960456048 .loss/FBt1/detection_loss‚4.534839081764221 _runtime‚1419.385893325 /loss/FRt1/assignment_loss‚4.708538667857647 /loss/FBt3/detection_loss‚0.7878612972443111 trainer/global_step‚1849 1loss/SRqqt2/assignment_loss‚2.500973025957743 1loss/FBt4/assignment_loss‚0.09598951414227486 .loss/FRt1/detection_loss‚2.522600916028023 1loss/SRqqt4/detection_loss‚1.8903526599376888 0loss/FRt4/assignment_loss‚3.9399496384386747 2loss/SRqqt4/assignment_loss‚1.7730132802310636 1loss/FBt3/assignment_loss‚0.24675713002512853 % _timestamp‚1.7884063163985136e+09 1loss/SRqqt3/assignment_loss‚1.713603460813983 0loss/FBt2/assignment_loss‚0.6675437791986951 .loss/FRt2/detection_loss‚2.508923080989293 /loss/FBt2/detection_loss‚2.13521911891607141;|7ZõÔ 0loss/FRt3/assignment_loss‚2.6088051199913025 /loss/FBt1/assignment_loss‚0.310475192964077 /loss/FRt4/detection_loss‚2.2391393921285307 %loss/total_loss‚55.77098063953024 epoch‚1 0loss/SRqqt1/detection_loss‚2.548379985672055 0loss/SRqqt2/detection_loss‚2.829245748122533 0loss/SRqqt3/detection_loss‚1.326656960456048 .loss/FBt1/detection_loss‚4.534839081764221 _runtime‚1419.385893325 /loss/FRt1/assignment_loss‚4.708538667857647 /loss/FBt3/detection_loss‚0.7878612972443111 trainer/global_step‚1849 1loss/SRqqt2/assignment_loss‚2.500973025957743 1loss/FBt4/assignment_loss‚0.09598951414227486 .loss/FRt1/detection_loss‚2.522600916028023 1loss/SRqqt4/detection_loss‚1.8903526599376888 0loss/FRt4/assignment_loss‚3.9399496384386747 2loss/SRqqt4/assignment_loss‚1.7730132802310636 1loss/FBt3/assignment_loss‚0.24675713002512853 % _timestamp‚1.7884063163985136e+09 1loss/SRqqt3/assignment_loss‚1.713603460813983 0loss/FBt2/assignment_loss‚0.6675437791986951 .loss/FRt2/detection_loss‚2.508923080989293 /loss/FBt2/detection_loss‚2.1352191189160714 _step‚150 /loss/FRt2/assignment_loss‚5.336807693753924 1loss/SRqqt1/assignment_loss‚2.465936025886825 /loss/FRt3/detection_loss‚1.8066625259086198 /loss/FBt4/detection_loss‚4.2727073431015015–Ä0˜»’ö‚(:‡ ²ÔãÔжœ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03mŸþ ÷‚(:½ ²ÔãÔ­àì  diskÐ÷r(.nvme0n1.in‚8.34375" disk.dm-0.out‚3755.52783203125 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd0.in‚0$ memory_percent‚99.95980262756348# proc.memory.rssMB‚ 2814.41015625 network.recv‚563087* disk./.usagePercent‚15.912344392476507 disk.sda3.out‚ 64.16796875 disk.drbd2080.out‚0 disk.sda1.in‚0* proc.memory.percent‚17.177796363830566 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 64.16796875 disk.drbd2265.in‚0' disk.nvme0c0n1.out‚3755.52783203125 disk.drbd1716.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚3755.52783203125 disk.dm-0.in‚8.34375 disk.sda.out‚ 64.16796875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚0% proc.memory.availableMB‚ 6.5859375 network.sent‚ 3.115552e+06 disk.rbd1.out‚ 7.966796875 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd2266.out‚0 cpu‚25.26515674877339% disk./.usageGB‚237.10543060302734w«¿’ø‚(:‡ ÁÔãÔª® gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ÓÿzCù‚(:¸ ÁÔãÔʾ‚ * disk./.usagePercent‚15.912341832401166$ disk./.usageGB‚237.1053924560547 disk.sda.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚17.177796363830566 network.recv‚569354 disk.sda3.out‚ 64.8046875 disk.rbd1.out‚ 8.052734375 disk.nvme0n1.in‚8.34375 disk.sda.out‚ 64.8046875 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 network.sent‚ 3.224032e+06 disk.drbd1716.in‚0% disk.nvme0n1.out‚3756.89013671875 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.availableMB‚6.78125# proc.memory.rssMB‚ 2814.41015625 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚8.34375 disk.dm-1.out‚ 64.8046875 cpu‚25.262777006971234' disk.nvme0c0n1.out‚3756.89013671875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd2266.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0$ memory_percent‚99.95861053466797" disk.dm-0.out‚3756.890136718758¯ó!2ú- _wandbruntime‚1454  _runtime‚1454ÚóørÉûà % _timestamp‚1.7884063519060612e+09 _runtime‚1454.893498404 _step‚151 ' lr-AdamW/pg1‚0.00027233892133598917 ' lr-AdamW/pg2‚0.00027233892133598917 trainer/global_step‚1899fÊK´ÎüÈ _runtime‚1454.893498404 _step‚151 ' lr-AdamW/pg1‚0.00027233892133598917 ' lr-AdamW/pg2‚0.00027233892133598917 trainer/global_step‚1899 % _timestamp‚1.7884063519060612e+09—Ãn™;2ý- _wandbruntime‚1454  _runtime‚1454Ñy ×'þÑO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27318774 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15934652 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108132125083215 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1574415305947 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121203627674112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1580229679104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35646360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27318776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15934654 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108131453606107 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1580886040576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109407960108544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110988189787648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109407978111488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24974102 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884063519068334e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119543204143616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35646360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1580229679104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1575069743695 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27318774 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106557038300160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24973660 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1660227576832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35666396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9039008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121203431720448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110988189787648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27318776 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35666396 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106557055339520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109407978111488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1660399590400 trainer/global_step‚1899 _runtime‚1454.894517817 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1660399590400 trainer/global_step‚1899 _runtime‚1454.894517817 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1580229679104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35646360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27318776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15934654 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108131453606107 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1580886040576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109407960108544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110988189787648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109407978111488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24974102 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884063519068334e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119543204143616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35646360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1580229679104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1575069743695 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27318774 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106557038300160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24973660 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1660227576832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35666396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9039008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121203431720448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110988189787648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27318776 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35666396 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106557055339520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109407978111488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1581049106944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1661248327168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9044029 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1581705481728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108187845648947 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24986389 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35684876 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1581705481728 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15942820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119604854881280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106611959470080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35664814 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121265933305344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111046036632064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111046036632064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109464313147392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35684876 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121266127148544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109464331150336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1581049106944 trainer/global_step‚1899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27332792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.alloÔ;¸ýcated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109464331150336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1575231740911 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62997604 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9043569 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108187174171631 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63017668 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27332790 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111045362254336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚1455.568959812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106611942430720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62997604 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1575886178867 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27332790 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24986851 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169903104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844ç«æ&„àM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1581705481728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108187845648947 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24986389 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35684876 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1581705481728 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15942820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119604854881280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106611959470080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35664814 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121265933305344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111046036632064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111046036632064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109464313147392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35684876 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121266127148544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109464331150336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1581049106944 trainer/global_step‚1899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27332792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109464331150336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1575231740911 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62997604 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9043569 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108187174171631 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63017668 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27332790 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111045362254336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚1455.568959812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106611942430720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62997604 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1575886178867 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27332790 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24986851 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169903104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1581049106944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1661248327168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9044029 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0™ ýrÈ2…- _wandbruntime‚1455  _runtime‚1455!ÎàŸZ†Ô /loss/FRt3/detection_loss‚2.1535064517412077 /loss/FRt4/detection_loss‚2.5823017236861316 1loss/SRqqt1/detection_loss‚3.0055604303876557 /loss/FRt2/assignment_loss‚4.435763155159197 2loss/SRqqt4/assignment_loss‚1.9067557585590027 /loss/FBt3/detection_loss‚0.8743948457388985 epoch‚1 % _timestamp‚1.7884063525826373e+09 0loss/FRt1/assignment_loss‚4.1710431687533855 /loss/FRt3/assignment_loss‚2.214367083338804 1loss/FBt4/assignment_loss‚0.10443964181467891 /loss/FBt1/detection_loss‚2.6686164140701294 0loss/FRt4/assignment_loss‚3.7955228957262905 0loss/FBt2/assignment_loss‚0.7623118087649345 1loss/SRqqt4/detection_loss‚1.5042069335194195 /loss/FBt4/detection_loss‚3.5850341830934798 trainer/global_step‚1899 _step‚154 .loss/FRt1/detection_loss‚2.810491230338812 _runtime‚1455.570002338 2loss/SRqqt1/assignment_loss‚2.5512385616699853 0loss/FBt1/assignment_loss‚0.5795724714782106 .loss/FRt2/detection_loss‚2.335965959649337 1loss/SRqqt2/assignment_loss‚1.889081697638442 2loss/SRqqt3/assignment_loss‚1.6942619931484972 1loss/FBt3/assignment_loss‚0.20774532372266508 1loss/SRqqt2/detection_loss‚1.9635921883873824 0loss/SRqqt3/detection_loss‚1.459977942651936 .loss/FBt2/detection_loss‚2.675730012357235 %loss/total_loss‚51.93148187539572gÁ‚_‡Ù /loss/FRt2/assignment_loss‚4.435763155159197 2loss/SRqqt4/assignment_loss‚1.9067557585590027 /loss/FBt3/detection_loss‚0.8743948457388985 epoch‚1 % _timestamp‚1.7884063525826373e+09 0loss/FRt1/assignment_loss‚4.1710431687533855 /loss/FRt3/assignment_loss‚2.214367083338804 1loss/FBt4/assignment_loss‚0.10443964181467891 /loss/FBt1/detection_loss‚2.6686164140701294 0loss/FRt4/assignment_loss‚3.7955228957262905 0loss/FBt2/assignment_loss‚0.7623118087649345 1loss/SRqqt4/detection_loss‚1.5042069335194195 /loss/FBt4/detection_loss‚3.5850341830934798 trainer/global_step‚1899 _step‚154 .loss/FRt1/detection_loss‚2.810491230338812 _runtime‚1455.570002338 2loss/SRqqt1/assignment_loss‚2.5512385616699853 0loss/FBt1/assignment_loss‚0.5795724714782106 .loss/FRt2/detection_loss‚2.335965959649337 1loss/SRqqt2/assignment_loss‚1.889081697638442 2loss/SRqqt3/assignment_loss‚1.6942619931484972 1loss/FBt3/assignment_loss‚0.20774532372266508 1loss/SRqqt2/detection_loss‚1.9635921883873824 0loss/SRqqt3/detection_loss‚1.459977942651936 .loss/FBt2/detection_loss‚2.675730012357235 %loss/total_loss‚51.93148187539572 /loss/FRt3/detection_loss‚2.1535064517412077 /loss/FRt4/detection_loss‚2.5823017236861316 1loss/SRqqt1/detection_loss‚3.0055604303876557št t‘ˆ‚(:† ßÔãÔÿ•Ö gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚185.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å&iLG‰‚(:¼ ßÔãÔøù’ % disk.nvme0n1.out‚3757.49267578125 disk.sda.out‚ 65.80078125 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚3757.49267578125# proc.memory.rssMB‚ 2814.41015625 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚8.34375$ disk./.usageGB‚237.1050910949707 disk.rbd1.in‚0 disk.rbd1.out‚ 8.115234375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17* proc.memory.percent‚17.177796363830566 disk.dm-0.in‚8.34375" disk.dm-0.out‚3757.49267578125 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0$ memory_percent‚99.96469020843506 network.sent‚ 3.326347e+06 disk.nvme0n1.in‚8.34375 disk.drbd2266.out‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 5.78515625 cpu‚25.25850278159614 network.recv‚579249 disk.drbd1716.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 65.80078125 disk.sda3.out‚ 65.80078125) disk./.usagePercent‚15.91232160780596 disk.drbd1716.in‚0 disk.drbd2080.in‚0'QÑ’Š‚(:‡ îÔãÔ¤§¶ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.017' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UF‹‚(:» îÔãÔäθ  network.recv‚585233 disk.rbd0.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚3757.74267578125 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 3.424152e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0$ memory_percent‚99.96273517608643 disk.drbd2266.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2814.41015625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 66.2890625 disk.drbd2266.out‚0* proc.memory.percent‚17.177796363830566) disk./.usagePercent‚15.91232160780596 disk.sda.out‚ 66.2890625 disk.drbd2080.in‚0 disk.nvme0n1.in‚8.34375& proc.memory.availableMB‚ 6.10546875 cpu‚25.256255624329338$ disk./.usageGB‚237.1050910949707 disk.sda1.in‚0 disk.nvme0c0n1.in‚8.34375 disk.rbd1.out‚ 8.2568359375 disk.dm-0.in‚8.34375" disk.dm-0.out‚3757.74267578125% disk.nvme0n1.out‚3757.74267578125 disk.dm-1.out‚ 66.2890625|Ù“&2Œ- _wandbruntime‚1490  _runtime‚1490¤ìùÆÀ _runtime‚1490.830059919 _step‚155 & lr-AdamW/pg1‚0.0002719319711147221 & lr-AdamW/pg2‚0.0002719319711147221 trainer/global_step‚1949 $ _timestamp‚1.788406387842628e+09µ·b´ËŽÅ $ _timestamp‚1.788406387842628e+09 _runtime‚1490.830059919 _step‚155 & lr-AdamW/pg1‚0.0002719319711147221 & lr-AdamW/pg2‚0.0002719319711147221 trainer/global_step‚1949›® l2- _wandbruntime‚1490  _runtime‚1490•%]ž×'ÑO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28019574 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9268446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1621215484928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64612024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28019576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16343052 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122625741026816 trainer/global_step‚1949 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64591986 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112225630053888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1621215484928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28019576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113846827535872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _runtime‚1490.831006605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124329030050304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1703456842752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112225612050944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1615250645355 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113847501900288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36572412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16343054 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110918166950623 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1621871846400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚200382‹DM‘ÖO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28019576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16343052 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122625741026816 trainer/global_step‚1949 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64591986 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112225630053888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1621215484928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28019576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113846827535872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _runtime‚1490.831006605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124329030050304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1703456842752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112225612050944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1615250645355 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113847501900288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36572412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16343054 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110918166950623 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1621871846400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28019574 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9268446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1621215484928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64612024œ'™˜2’- _wandbruntime‚1491  _runtime‚1491⧀Lá&“ÛM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64624456 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64624456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109357165998080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9273046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16351220 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1616067080319 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122687391764480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1704323274240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109357148958720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25624732 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113904674380288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113904000002560 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64644520 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36610928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155223040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚1491.514358926 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110973216039039 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28033590 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124391559815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28033592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112281965089792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112281983092736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1622034912768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124391738978816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚1949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113904674380288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1622691287552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112281965089792 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25624266 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122687415704576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36590866 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28033592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110973887516355 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112281983092736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36590866 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1622691287552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1616721518275 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1622034912768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179163136 % _timestamp‚1.7884063885263598e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9273510 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112281983092736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1622034912768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124391738978816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚1949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113904674380288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1622691287552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112281965089792 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25624266 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122687415704576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36590866 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28033592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110973887516355 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112281983092736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36590866 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1622691287552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1616721518275 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1622034912768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179163136 % _timestamp‚1.7884063885263598e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9273510 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9273046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16351220 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1616067080319 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122687391764480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1704323274240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109357148958720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25624732 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113904674380288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113904000002560 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64644520 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36610928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155223040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚1491.514358926 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110973216039039 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944ÜLó‚2•- _wandbruntime‚1491  _runtime‚1491Á„1‹Y–Ó 1loss/SRqqt3/assignment_loss‚1.351656782832977 /loss/FBt3/detection_loss‚0.9270422787510624 2loss/SRqqt2/assignment_loss‚2.0467199277208774 2loss/FBt4/assignment_loss‚0.052626880312648915 /loss/FRt3/detection_loss‚1.9682270726561546 /loss/SRqqt2/detection_loss‚2.05895781699492 0loss/SRqqt4/detection_loss‚2.343395345223447 0loss/FRt2/assignment_loss‚3.7843211889266968 2loss/SRqqt1/assignment_loss‚2.5231651320601953 /loss/FRt4/detection_loss‚2.3127486876079013 1loss/SRqqt1/detection_loss‚2.6759065371571165 trainer/global_step‚1949 /loss/FRt2/detection_loss‚2.5487064609924954 /loss/FRt3/assignment_loss‚2.066363466978073 /loss/FRt4/assignment_loss‚3.847713393824441 0loss/FBt1/assignment_loss‚0.3783742979874686 0loss/SRqqt3/detection_loss‚1.202450246010956 _runtime‚1491.515738418 /loss/FRt1/assignment_loss‚4.525409238679068 0loss/FBt2/assignment_loss‚0.6937844587696923 &loss/total_loss‚51.888639254592164 1loss/FBt3/assignment_loss‚0.17524163706465817 /loss/FBt1/detection_loss‚2.9261428874272566 .loss/FBt2/detection_loss‚2.535386650098695 .loss/FBt4/detection_loss‚4.282250205675761 epoch‚1 % _timestamp‚1.7884063885280678e+09 2loss/SRqqt4/assignment_loss‚1.7181576273093622 /loss/FRt1/detection_loss‚2.9438910335302353 _step‚158 %^—Ø 2loss/SRqqt2/assignment_loss‚2.0467199277208774 2loss/FBt4/assignment_loss‚0.052626880312648915 /loss/FRt3/detection_loss‚1.9682270726561546 /loss/SRqqt2/detection_loss‚2.05895781699492 0loss/SRqqt4/detection_loss‚2.343395345223447 0loss/FRt2/assignment_loss‚3.7843211889266968 2loss/SRqqt1/assignment_loss‚2.5231651320601953 /loss/FRt4/detection_loss‚2.3127486876079013 1loss/SRqqt1/detection_loss‚2.6759065371571165 trainer/global_step‚1949 /loss/FRt2/detection_loss‚2.5487064609924954 /loss/FRt3/assignment_loss‚2.066363466978073 /loss/FRt4/assignment_loss‚3.847713393824441 0loss/FBt1/assignment_loss‚0.3783742979874686 0loss/SRqqt3/detection_loss‚1.202450246010956 _runtime‚1491.515738418 /loss/FRt1/assignment_loss‚4.525409238679068 0loss/FBt2/assignment_loss‚0.6937844587696923 &loss/total_loss‚51.888639254592164 1loss/FBt3/assignment_loss‚0.17524163706465817 /loss/FBt1/detection_loss‚2.9261428874272566 .loss/FBt2/detection_loss‚2.535386650098695 .loss/FBt4/detection_loss‚4.282250205675761 epoch‚1 % _timestamp‚1.7884063885280678e+09 2loss/SRqqt4/assignment_loss‚1.7181576273093622 /loss/FRt1/detection_loss‚2.9438910335302353 _step‚158 1loss/SRqqt3/assignment_loss‚1.351656782832977 /loss/FBt3/detection_loss‚0.9270422787510624ž#$þœ‘˜‚(:† ýÔãÔ¬¼é gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.975' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07úÖm3™‚(:¨ ýÔãÔÙÀ™  disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 6.21484375* proc.memory.percent‚17.177796363830566 disk.nvme0n1.out‚ 3758.09375 disk.sda.out‚ 66.8984375 disk.drbd2080.in‚0 network.recv‚589624 disk.rbd1.in‚0 disk.rbd1.out‚ 8.2568359375 disk.nvme0n1.in‚8.34375 disk.sda3.out‚ 66.8984375 cpu‚25.25439940819223! disk.nvme0c0n1.out‚ 3758.09375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚8.34375 disk.dm-1.in‚0 disk.dm-1.out‚ 66.8984375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.out‚ 3758.09375 disk.drbd2265.in‚0 disk.sda3.in‚0$ memory_percent‚99.96206760406494* disk./.usagePercent‚15.912330056054591 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2814.41015625 network.sent‚ 3.4304e+06% disk./.usageGB‚237.105216979980476¦E~š‚(:’ ŒÕãÔêÏ· gpu.0.gpu‚12.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.007' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.401400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Ú“H›‚(:½ ŒÕãÔÁª“  disk.drbd2080.out‚0 network.sent‚ 3.537984e+06 disk.dm-1.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚8.34375 disk.sda.out‚ 67.3828125 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚8.34375 disk.drbd2265.in‚0 disk.drbd2266.in‚0& proc.memory.availableMB‚ 6.18359375# proc.memory.rssMB‚ 2814.41015625* proc.memory.percent‚17.177796363830566' disk.nvme0c0n1.out‚3759.05615234375 disk.drbd2265.out‚0 disk.sda3.out‚ 67.3828125$ memory_percent‚99.96225833892822 cpu‚25.252247910038925 disk.rbd1.in‚0% disk.nvme0n1.out‚3759.05615234375* disk./.usagePercent‚15.912330056054591 disk.sda2.in‚0 disk.sda.in‚0% disk./.usageGB‚237.10521697998047 disk.drbd1716.out‚0 disk.rbd1.out‚ 8.3193359375 network.recv‚595284 disk.sda1.in‚0 disk.nvme0c0n1.in‚8.34375" disk.dm-0.out‚3759.05615234375 disk.dm-1.out‚ 67.3828125Hažx2œ- _wandbruntime‚1526  _runtime‚1526£n£¦ÆÀ _runtime‚ 1526.75133281 _step‚159 & lr-AdamW/pg1‚0.0002714970116251128 & lr-AdamW/pg2‚0.0002714970116251128 trainer/global_step‚1999 % _timestamp‚1.7884064237638817e+09fŠü˞Š& lr-AdamW/pg1‚0.0002714970116251128 & lr-AdamW/pg2‚0.0002714970116251128 trainer/global_step‚1999 % _timestamp‚1.7884064237638817e+09 _runtime‚ 1526.75133281 _step‚159Ÿ<22Ÿ- _wandbruntime‚1526  _runtime‚1526ÁDr ÑO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_startEFIV'/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116706139648512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1656085984763 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9498795 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125708277910016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66238876 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26249805 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 trainer/global_step‚1999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37518500 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127454923034112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115043263993344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1746451267584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1656740422511 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16751454 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116705465284096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1662201290752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1746621184000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113703537340923 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115043281996288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115043263993344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28720374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116705465284096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28720374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26250249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28720376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112047468395520 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115043263993344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1746451267584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1656740422511 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16751454 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116705465284096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1662201290752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1746621184000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113703537340923 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115043281996288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115043263993344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28720374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116705465284096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28720374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26250249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28720376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112047468395520 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26263001 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116763312128512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116763312128512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1657556857683 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1656902419727 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28734390 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1747479242240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 trainer/global_step‚1999 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28734392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 % _timestamp‚1.7884064244470115e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66251308 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26262549 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37516918 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37536980 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113759929383763 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116762637750784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1663020718592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125769952587776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115099635035136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115099617032192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16759622 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28734390?¿+™ ¤ßM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115099617032192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16759622 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28734390 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16759620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1663677093376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28734392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26263001 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116763312128512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116763312128512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1657556857683 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1656902419727 ?5DeviceSta8jÁLtsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28734390 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1747479242240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 trainer/global_step‚1999 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28734392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 % _timestamp‚1.7884064244470115e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66251308 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26262549 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37516918 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37536980 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113759929383763 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116762637750784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1663020718592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125769952587776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115099635035136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450¡SJð62¥- _wandbruntime‚1527  _runtime‚15270Á³S¦Í _runtime‚ 1527.4364235 1loss/FBt3/assignment_loss‚0.25023944499887024 /loss/FBt4/assignment_loss‚0.257524436712265 %loss/total_loss‚52.44442157790313 /loss/FRt1/assignment_loss‚4.588641105554043 /loss/FRt2/assignment_loss‚5.003274847160686 /loss/FRt4/assignment_loss‚4.400179002343155 1loss/SRqqt2/assignment_loss‚2.518885082254807 1loss/FBt2/assignment_loss‚0.45925521990284324 1loss/SRqqt3/detection_loss‚1.3622243376166532 _step‚162 0loss/FBt1/assignment_loss‚0.4205155150197082 1loss/SRqqt4/detection_loss‚1.7251748442649841 /loss/FRt1/detection_loss‚2.6512886652579675 /loss/FRt4/detection_loss‚2.6330477021089416 2loss/SRqqt3/assignment_loss‚1.6809991738109877 2loss/SRqqt4/assignment_loss‚1.7164337769150735 .loss/FRt2/detection_loss‚2.357593234289776 .loss/FRt3/detection_loss‚1.643668386484346 /loss/FBt4/detection_loss‚2.8787354946136476 0loss/FRt3/assignment_loss‚1.8209654659994188 epoch‚1 trainer/global_step‚1999 $ _timestamp‚1.788406424449049e+09 2loss/SRqqt1/assignment_loss‚2.5920834485441446 0loss/SRqqt1/detection_loss‚2.468661569058895 0loss/SRqqt2/detection_loss‚2.646408946977721 .loss/FBt1/detection_loss‚2.593540682512171 .loss/FBt2/detection_loss‚2.919221873084704 /loss/FBt3/detection_loss‚0.8558593224173212Ë;X§Ò /loss/FRt1/detection_loss‚2.6512886652579675 /loss/FRt4/detection_loss‚2.6330477021089416 2loss/SRqqt3/assignment_loss‚1.6809991738109877 2loss/SRqqt4/assignment_loss‚1.7164337769150735 .loss/FRt2/detection_loss‚2.357593234289776 .loss/FRt3/detection_loss‚1.643668386484346 /loss/FBt4/detection_loss‚2.8787354946136476 0loss/FRt3/assignment_loss‚1.8209654659994188 epoch‚1 trainer/global_step‚1999 $ _timestamp‚1.788406424449049e+09 2loss/SRqqt1/assignment_loss‚2.5920834485441446 0loss/SRqqt1/detection_loss‚2.468661569058895 0loss/SRqqt2/detection_loss‚2.646408946977721 .loss/FBt1/detection_loss‚2.593540682512171 .loss/FBt2/detection_loss‚2.919221873084704 /loss/FBt3/detection_loss‚0.8558593224173212 _runtime‚ 1527.4364235 1loss/FBt3/assignment_loss‚0.25023944499887024 /loss/FBt4/assignment_loss‚0.257524436712265 %loss/total_loss‚52.44442157790313 /loss/FRt1/assignment_loss‚4.588641105554043 /loss/FRt2/assignment_loss‚5.003274847160686 /loss/FRt4/assignment_loss‚4.400179002343155 1loss/SRqqt2/assignment_loss‚2.518885082254807 1loss/FBt2/assignment_loss‚0.45925521990284324 1loss/SRqqt3/detection_loss‚1.3622243376166532 _step‚162 0loss/FBt1/assignment_loss‚0.4205155150197082 1loss/SRqqt4/detection_loss‚1.7251748442649841¢|gØ«¨‚(:’ ›ÕãÔ决 gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.596' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.919200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YjÙ:©‚(:¯ ›ÕãÔé¨ý  disk.rbd1.out‚ 8.3193359375 disk.sda.out‚67.9375 disk.nvme0c0n1.in‚8.34375 disk.rbd1.in‚0 disk.dm-1.out‚67.9375 disk.drbd2265.out‚0 network.sent‚ 3.543876e+06* disk./.usagePercent‚15.912340808371027% disk./.usageGB‚237.10537719726562 disk.drbd1716.out‚0% disk.nvme0n1.out‚3759.30224609375$ memory_percent‚99.96223449707031* proc.memory.percent‚17.177796363830566 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0" proc.memory.availableMB‚6.1875 network.recv‚599433 disk.sda3.out‚67.9375 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.dm-0.in‚8.34375# proc.memory.rssMB‚ 2814.41015625 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚8.34375" disk.dm-0.out‚3759.30224609375 disk.dm-1.in‚0 disk.sda.in‚0 cpu‚25.25030283006088 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚3759.30224609375t¬ã’ª‚(:‡ ªÕãÔ皸 gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×_¼£,«‚(:¡ ªÕãÔ“Ïø  disk.nvme0c0n1.in‚8.34375 disk.drbd1716.in‚0 disk.nvme0n1.out‚3759.75* proc.memory.percent‚17.177796363830566 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 8.5234375 disk.nvme0n1.in‚8.34375 disk.dm-0.in‚8.34375 disk.drbd2080.in‚0# memory_percent‚99.9619722366333 network.recv‚605577% disk./.usageGB‚237.10530471801758 disk.sda.out‚ 68.41015625 disk.sda3.in‚0 disk.sda3.out‚ 68.41015625 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.out‚3759.75 disk.dm-1.out‚ 68.41015625 disk.drbd2265.in‚0 cpu‚25.248390316065155 disk.drbd2266.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 6.23046875 proc.cpu.threads‚17* disk./.usagePercent‚15.912335944227879 disk.sda1.out‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2814.41015625 disk.dm-0.out‚3759.75 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 network.sent‚ 3.651967e+06xÑb’¬‚(:‡ ¹ÕãÔåЕ  gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.211' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j†þ0E­‚(:º ¹ÕãÔ™ÐÐ  disk.sda1.in‚0 disk.rbd1.out‚8.609375% disk./.usageGB‚237.10739517211914 disk.rbd0.in‚0" disk.dm-0.out‚3767.96630859375 disk.sda.in‚0 cpu‚25.246299322290813 network.sent‚ 3.65826e+06 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375' disk.nvme0c0n1.out‚3767.96630859375 disk.dm-0.in‚8.34375 disk.drbd2265.in‚0 disk.sda.out‚ 72.37890625# proc.memory.rssMB‚ 2814.41015625* disk./.usagePercent‚15.912476236356643 disk.drbd2266.out‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚3767.96630859375* proc.memory.percent‚17.177796363830566 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 72.37890625 disk.drbd2265.out‚0 disk.sda3.out‚ 72.37890625 disk.drbd2080.out‚0# memory_percent‚99.9626874923706& proc.memory.availableMB‚ 6.11328125 proc.cpu.threads‚17 network.recv‚610181 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚8.34375 disk.dm-1.in‚0 disk.drbd2080.in‚0µûð2®- _wandbruntime‚1562  _runtime‚1562œ[-ů¿ trainer/global_step‚2049 % _timestamp‚1.7884064598682122e+09 _runtime‚1562.855641425 _step‚163 % lr-AdamW/pg1‚0.000271034133512067 % lr-AdamW/pg2‚0.000271034133512067’kÑÊ°Ä trainer/global_step‚2049 % _timestamp‚1.7884064598682122e+09 _runtime‚1562.855641425 _step‚163 % lr-AdamW/pg1‚0.000271034133512067 % lr-AdamW/pg2‚0.000271034133512067£ŽÅÉÓ2±- _wandbruntime‚1562  _runtime‚1562ñ…„Ö'²ÐO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1703187096576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116489579208331 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38424516 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130580569456640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67865728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38424516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26887793 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38444552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17159854 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1697575761919 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117860915935744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9728384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9727941 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153139200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29421176 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26888238 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116490250685439 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119564777396736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580392377344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 1562.85658822 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119564103032320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17159852 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119564103032320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117860933938688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117860915935744 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884064598689423e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1703187096576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 trainer/global_step‚2049 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1789577584128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29421174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29421174 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1703843458048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1696921324171¬œþ2³ÕO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1697575761919 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117860915935744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9728384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9727941 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153139200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29421176 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26888238 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116490250685439 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119564777396736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580392377344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 1562.85658822 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119564103032320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17159852 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119564103032320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117860933938688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117860915935744 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884064598689423e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1703187096576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.currexO© nt‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 trainer/global_step‚2049 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1789577584128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29421174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29421174 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1703843458048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1696921324171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1703187096576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116489579208331 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38424516 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130580569456640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67865728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38424516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26887793 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38444552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17159854 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0¤ÂW2´- _wandbruntime‚1563  _runtime‚1563í8ë÷á&µÛM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1697737759135 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67898224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚1563.539030617 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119621949876736 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67878160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117917268974592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9732832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884064605511146e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1698392197091 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116545971251171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117917286977536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1790595271680 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38463032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128852465530880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚2049 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26900402 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117917286977536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114847579054080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 _step‚165 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38442970 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130642888802304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128852489470976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67898224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚1563.539030617 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119621949876736 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67878160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117917268974592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9732832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884064605511146e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1698392197091 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116545971251171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117917286977536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1790595271680 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38463032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128852465530880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚2049 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26900402 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117917286977536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114847579054080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 _step‚165 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38442970 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130642888802304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128852489470976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39370604 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1738411101327 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744172902400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131ìà,i873351676416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚2099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744172902400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122422740780544 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9957688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69472542 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122422740780544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1744829263872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1832864774656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27525940 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133706048631808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1832696955392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9958143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119276292552847 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69492580 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117537864412160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17568252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30121974 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744829263872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1737756663579 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39370604 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27526397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117537881451520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Þ§-ìÜ'ÃÖO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120678567878144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133706240391168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39350568 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122423415144960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122423415144960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120678585881088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120678585881088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120678567878144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119275621075739 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚1598.709844455 _step‚168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39370604 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1738411101327 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744172902400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131873351676416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚2099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744172902400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122422740780544 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9957688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69472542 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122422740780544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1744829263872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1832864774656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27525940 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133706048631808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1832696955392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9958143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119276292552847 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69492580 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117537864412160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17568252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30121974 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744829263872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1737756663579 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39370604 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27526397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117537881451520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30121976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39350568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605¨ò”’2Ä- _wandbruntime‚1599  _runtime‚1599·,á&ÅÛM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131935002414080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119331341641263 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122479913247232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9962706 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30135992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120734920916992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131935026354176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120734938919936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884064963865662e+09 _step‚169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745648705024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27538664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17576420 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122480587624960 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30135992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133768746740736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117592785582080 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120734920916992 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9962706 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30135992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120734920916992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131935026354176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120734938919936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884064963865662e+09 _step‚169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745648705024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27538664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17576420 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122480587624960 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30135992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133768746740736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117592785582080 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944©†ã U2Ç- _wandbruntime‚1599  _runtime‚1599ÙlãåYÈÓ .loss/FRt2/detection_loss‚2.544287065664927 0loss/FRt1/assignment_loss‚4.2790222276340835 /loss/FRt2/assignment_loss‚4.308295293287798 0loss/FRt3/assignment_loss‚2.4016351079454226 2loss/SRqqt3/assignment_loss‚1.4554430077550933 1loss/SRqqt3/detection_loss‚1.2393693902995437 trainer/global_step‚2099 2loss/SRqqt4/assignment_loss‚1.5731919256218694 % _timestamp‚1.7884064963880486e+09 _step‚170 2loss/SRqqt1/assignment_loss‚2.1410264320233288 epoch‚1 0loss/FBt1/assignment_loss‚0.5696224165814263 .loss/FRt1/detection_loss‚2.649402678012848 _runtime‚1599.375402964 1loss/FBt4/assignment_loss‚0.23614863554636636 2loss/SRqqt2/assignment_loss‚2.1462586039194353 1loss/SRqqt2/detection_loss‚1.6790758575554248 /loss/FRt3/detection_loss‚1.8361350845317452 0loss/SRqqt4/detection_loss‚1.856299113024745 .loss/FBt1/detection_loss‚3.224317056792123 .loss/FBt4/detection_loss‚4.563552379608154 0loss/FRt4/assignment_loss‚3.4741992658498337 1loss/FBt3/assignment_loss‚0.24786098141199514 0loss/SRqqt1/detection_loss‚2.614114272243836 .loss/FBt2/detection_loss‚2.876914207140605 /loss/FBt3/detection_loss‚0.7269271859302535 %loss/total_loss‚51.59768358054922 0loss/FBt2/assignment_loss‚0.3956669610712197 /loss/FRt4/detection_loss‚2.5589184310971476­GÕ5^ÉØ 2loss/SRqqt4/assignment_loss‚1.5731919256218694 % _timestamp‚1.7884064963880486e+09 _step‚170 2loss/SRqqt1/assignment_loss‚2.1410264320233288 epoch‚1 0loss/FBt1/assignment_loss‚0.5696224165814263 .loss/FRt1/detection_loss‚2.649402678012848 _runtime‚1599.375402964 1loss/FBt4/assignment_loss‚0.23614863554636636 2loss/SRqqt2/assignment_loss‚2.1462586039194353 1loss/SRqqt2/detection_loss‚1.6790758575554248 /loss/FRt3/detection_loss‚1.8361350845317452 0loss/SRqqt4/detection_loss‚1.856299113024745 .loss/FBt1/detection_loss‚3.224317056792123 .loss/FBt4/detection_loss‚4.563552379608154 0loss/FRt4/assignment_loss‚3.4741992658498337 1loss/FBt3/assignment_loss‚0.24786098141199514 0loss/SRqqt1/detection_loss‚2.614114272243836 .loss/FBt2/detection_loss‚2.876914207140605 /loss/FBt3/detection_loss‚0.7269271859302535 %loss/total_loss‚51.59768358054922 0loss/FBt2/assignment_loss‚0.3956669610712197 /loss/FRt4/detection_loss‚2.5589184310971476 .loss/FRt2/detection_loss‚2.544287065664927 0loss/FRt1/assignment_loss‚4.2790222276340835 /loss/FRt2/assignment_loss‚4.308295293287798 0loss/FRt3/assignment_loss‚2.4016351079454226 2loss/SRqqt3/assignment_loss‚1.4554430077550933 1loss/SRqqt3/detection_loss‚1.2393693902995437 trainer/global_step‚2099ª‡ñÃ&’Ê‚(:‡ æÕãÔ¼©± gpu.0.gpu‚50.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÃû–BË‚(:· æÕãÔ¾øù  disk.drbd1716.out‚0 disk.rbd1.out‚ 8.8994140625 disk.nvme0n1.in‚8.34375* disk./.usagePercent‚15.912500301064862 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 74.23828125" proc.memory.rssMB‚ 2814.4140625 cpu‚25.240934129369183 disk.sda2.out‚0$ memory_percent‚99.96352195739746 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0% proc.memory.availableMB‚ 5.9765625 network.recv‚624202 disk.drbd2266.in‚0 disk.dm-1.out‚ 74.23828125 proc.cpu.threads‚17 network.sent‚ 3.777658e+06 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 74.23828125 disk.drbd2080.in‚0# disk.nvme0n1.out‚3772.873046875 disk.dm-0.in‚8.34375 disk.dm-0.out‚3772.873046875 disk.sda.in‚0 disk.sda2.in‚0% disk.nvme0c0n1.out‚3772.873046875* proc.memory.percent‚17.177820205688477$ disk./.usageGB‚237.1077537536621 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚8.34375™õÎÌ‚(:’ õÕãÔóÜ« gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚142.106' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.421199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u8’€AÍ‚(:¶ õÕãÔ·ùé  disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 75.06640625 disk.drbd2266.in‚0 cpu‚25.23919704487265 disk.rbd1.out‚ 8.9931640625 disk.drbd2266.out‚0$ proc.memory.availableMB‚5.734375 proc.cpu.threads‚17 network.sent‚ 3.885602e+06* disk./.usagePercent‚15.912505933230614 disk.drbd2080.in‚0% disk.nvme0c0n1.out‚3773.060546875 disk.drbd1716.out‚0 disk.dm-0.out‚3773.060546875 disk.dm-0.in‚8.34375 disk.drbd2265.in‚0 network.recv‚630609 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd1716.in‚0# disk.nvme0n1.out‚3773.060546875" proc.memory.rssMB‚ 2814.4140625 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 75.06640625$ memory_percent‚99.96500015258789* proc.memory.percent‚17.177820205688477 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375 disk.drbd2265.out‚0 disk.sda.out‚ 75.06640625 disk.sda3.in‚0% disk./.usageGB‚237.10783767700195 disk.rbd0.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0(³³2Î- _wandbruntime‚1634  _runtime‚1634bž€2ÉÏà ' lr-AdamW/pg1‚0.00027002501306618243 ' lr-AdamW/pg2‚0.00027002501306618243 trainer/global_step‚2149 % _timestamp‚1.7884065318764744e+09 _runtime‚1634.863945751 _step‚171]_ ôÎÐÈ ' lr-AdamW/pg1‚0.00027002501306618243 ' lr-AdamW/pg2‚0.00027002501306618243 trainer/global_step‚2149 % _timestamp‚1.7884065318764744e+09 _runtime‚1634.863945751 _step‚171«›X2Ñ- _wandbruntime‚1634  _runtime‚1634e2}Ù'ÒÓO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1778592002987 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2149 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30822774 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876036723200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1779246440735 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123496237823488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 % _timestamp‚1.7884065318771923e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28164917 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40276620 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122061662943147 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1785815069696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30822774 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125282052893184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134955912499712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125281378528768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120283087979520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚172 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28164464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122062334420255 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125281378528768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123496219820544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120283070940160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1634.864980993 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125281378528768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120283087979520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚172 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28164464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122062334420255 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125281378528768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123496219820544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120283070940160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1634.864980993 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1779246440735 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123496237823488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 % _timestamp‚1.7884065318771923e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28164917 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40276620 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122061662943147 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1785815069696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30822774 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125282052893184¬Q5nêÔ‚(:… „ÖãÔÚë® gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚186.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v¬âïGÕ‚(:¼ „ÖãÔÐÆ  disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚8.34375 disk.rbd1.out‚ 9.0556640625 disk.drbd2266.in‚0* disk./.usagePercent‚15.912504653192943% disk./.usageGB‚237.10781860351562 disk.dm-1.out‚ 75.58203125 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 75.58203125" disk.dm-0.out‚3773.44677734375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.in‚0 cpu‚25.237433217291965 disk.rbd0.in‚0' disk.nvme0c0n1.out‚3773.44677734375 disk.rbd1.in‚0$ memory_percent‚99.96500015258789$ proc.memory.availableMB‚5.734375 network.sent‚ 3.89087e+06 disk.dm-1.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚3773.44677734375* proc.memory.percent‚17.177820205688477 network.recv‚634179 disk.sda.out‚ 75.58203125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚8.34375" proc.memory.rssMB‚ 2814.4140625 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚8.34375°’‹˜2Ö- _wandbruntime‚1635  _runtime‚1635<磒Ë×ÝM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71151928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.sÜÂQs&mall_pool.freed‚ 1785978136064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28177170 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125339225373184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17984820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1786634510848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135017563237376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123552590862336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123552572859392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28177634 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120337992110080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123552572859392 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884065325573132e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17984822 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30836792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 _runtime‚1635.545314804 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30836790 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1876708381696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176194560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1785978136064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122118054985987 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125338550995456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135017539297280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122117383508671 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1780062875907 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10192812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136894447813632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71131864 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚173 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30836792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40315136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136894247678976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123552590862336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28177170 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125339225373184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17984820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1786634510848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135017563237376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123552590862336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123552572859392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28177634 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120337992110080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123552572859392 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884065325573132e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17984822 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30836792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 _runtime‚1635.545314804 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30836790 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1876708381696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176194560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1785978136064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122118054985987 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125338550995456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135017539297280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122117383508671 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1780062875907 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10192812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136894447813632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71131864 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚173 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30836792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40315136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136894247678976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123552590862336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 B#2Ù- _wandbruntime‚1635  _runtime‚1635=öÇYÚÓ /loss/FRt2/detection_loss‚2.7185627275552506 /loss/FRt4/assignment_loss‚3.901771149635315 2loss/SRqqt3/assignment_loss‚1.9239031696613407 0loss/FBt1/assignment_loss‚0.6499496481627912 1loss/SRqqt3/detection_loss‚1.2103923567583863 /loss/FBt1/detection_loss‚2.9069831425493415 0loss/FRt1/assignment_loss‚4.2853023627909215 0loss/FBt2/assignment_loss‚0.7664325634638468 /loss/FRt1/detection_loss‚2.6520908838365136 .loss/FRt3/detection_loss‚1.944048228806683 /loss/FRt4/detection_loss‚2.4336644065380098 % _timestamp‚1.7884065325590281e+09 /loss/FBt3/assignment_loss‚0.335294296464513 .loss/FBt4/detection_loss‚3.732686916987101 2loss/SRqqt1/assignment_loss‚2.1321311311589346 1loss/FBt4/assignment_loss‚0.23857448001702627 1loss/SRqqt1/detection_loss‚2.3459352254867554 trainer/global_step‚2149 /loss/FRt2/assignment_loss‚4.418449469101735 2loss/SRqqt2/assignment_loss‚1.8396141946315765 1loss/SRqqt4/detection_loss‚1.6667083436340997 .loss/FBt2/detection_loss‚2.865148319138421 0loss/SRqqt2/detection_loss‚2.096839255756802 &loss/total_loss‚52.117754614393974 0loss/FRt3/assignment_loss‚2.3994282611778806 2loss/SRqqt4/assignment_loss‚1.9388006361144963 /loss/FBt3/detection_loss‚0.7150434449662307 _runtime‚ 1635.54639442 _step‚174 epoch‚1êa£^ÛØ 0loss/SRqqt2/detection_loss‚2.096839255756802 &loss/total_loss‚52.117754614393974 0loss/FRt3/assignment_loss‚2.3994282611778806 2loss/SRqqt4/assignment_loss‚1.9388006361144963 /loss/FBt3/detection_loss‚0.7150434449662307 _runtime‚ 1635.54639442 _step‚174 epoch‚1 /loss/FRt2/detection_loss‚2.7185627275552506 /loss/FRt4/assignment_loss‚3.901771149635315 2loss/SRqqt3/assignment_loss‚1.9239031696613407 0loss/FBt1/assignment_loss‚0.6499496481627912 1loss/SRqqt3/detection_loss‚1.2103923567583863 /loss/FBt1/detection_loss‚2.9069831425493415 0loss/FRt1/assignment_loss‚4.2853023627909215 0loss/FBt2/assignment_loss‚0.7664325634638468 /loss/FRt1/detection_loss‚2.6520908838365136 .loss/FRt3/detection_loss‚1.944048228806683 /loss/FRt4/detection_loss‚2.4336644065380098 % _timestamp‚1.7884065325590281e+09 /loss/FBt3/assignment_loss‚0.335294296464513 .loss/FBt4/detection_loss‚3.732686916987101 2loss/SRqqt1/assignment_loss‚2.1321311311589346 1loss/FBt4/assignment_loss‚0.23857448001702627 1loss/SRqqt1/detection_loss‚2.3459352254867554 trainer/global_step‚2149 /loss/FRt2/assignment_loss‚4.418449469101735 2loss/SRqqt2/assignment_loss‚1.8396141946315765 1loss/SRqqt4/detection_loss‚1.6667083436340997 .loss/FBt2/detection_loss‚2.865148319138421®Œ¼‡ Ü‚(:’ “ÖãÔƒ‡ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.065' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.012999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m°ßmFÝ‚(:» “ÖãÔùµå * disk./.usagePercent‚15.912560974850475 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.sda.in‚0 disk.drbd1716.out‚0" disk.dm-0.out‚3773.80224609375* proc.memory.percent‚17.177820205688477 disk.sda3.out‚ 76.1328125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚8.34375 disk.sda.out‚ 76.1328125$ memory_percent‚99.96545314788818 cpu‚25.23566039124183 proc.cpu.threads‚17 network.recv‚639715% disk./.usageGB‚237.10865783691406 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 76.1328125 disk.sda2.out‚0& proc.memory.availableMB‚ 5.66015625 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚8.34375 disk.rbd1.out‚ 9.0556640625% disk.nvme0n1.out‚3773.80224609375 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚3773.80224609375 disk.dm-0.in‚8.34375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2814.4140625 network.sent‚ 3.987762e+06YÈ“gÞ‚(:’ ¢ÖãÔÞ€å  gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.307' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.861399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07?ûWFß‚(:» ¢ÖãÔéÕä  disk.nvme0c0n1.in‚8.34375 disk.drbd2265.out‚0 disk.sda.out‚ 76.6328125$ memory_percent‚99.96240139007568 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.177820205688477 proc.cpu.threads‚17 disk.rbd1.out‚ 9.197265625% disk.nvme0n1.out‚3774.39501953125 disk.drbd2265.in‚0' disk.nvme0c0n1.out‚3774.39501953125 disk.rbd1.in‚0" disk.dm-0.out‚3774.39501953125 disk.sda.in‚0 cpu‚25.233950574770333 network.recv‚645841 network.sent‚ 4.085325e+06% disk./.usageGB‚237.10865783691406 disk.drbd1716.in‚0& proc.memory.availableMB‚ 6.16015625* disk./.usagePercent‚15.912560974850475 disk.drbd2080.in‚0 disk.dm-0.in‚8.34375 disk.dm-1.in‚0" proc.memory.rssMB‚ 2814.4140625 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚8.34375 disk.dm-1.out‚ 76.6328125 disk.sda3.out‚ 76.6328125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0±û}R2à- _wandbruntime‚1670  _runtime‚1670¿ÖC¦Éáà ' lr-AdamW/pg2‚0.00026947898103253204 trainer/global_step‚2199 % _timestamp‚1.7884065679096963e+09 _runtime‚1670.897190735 _step‚175 ' lr-AdamW/pg1‚0.00026947898103253204Í6Õ,ÎâÈ trainer/global_step‚2199 % _timestamp‚1.7884065679096963e+09 _runtime‚1670.897190735 _step‚175 ' lr-AdamW/pg1‚0.00026947898103253204 ' lr-AdamW/pg2‚0.00026947898103253204¯ÅŒä2ã- _wandbruntime‚1670  _runtime‚16705æfáôäÓO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31523576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126313871762944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚1670.898114869 _step‚176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41222708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72746284 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1919046320640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826144514048 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72726246 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826800875520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138038449382912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138038425442816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128140690641408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31523576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18385052 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918870112768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128140016276992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123028294507520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126313871762944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10415391 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28800887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128140016276992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884065679104695e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10415833 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124848376287663 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128140690641408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31523574 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚2199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126313889765888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31523574 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41202672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0i¡‘´Þ'åØO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31523576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18385052 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918870112768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128140016276992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123028294507520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126313871762944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10415391 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28800887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128140016276992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884065679104695e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10415833 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124848376287663 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128140690641408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31523574 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚2199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126313889765888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31523574 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41202672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31523576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126313871762944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚1670.898114869 _step‚176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41222708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72746284 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1919046320640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826144514048 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72726246 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826800875520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138038449382912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138038425442816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128140690641408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944°‰/Õ2æ- _wandbruntime‚1671  _runtime‚1671S(N¢ã&çÝM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10420405 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128197863121408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128197188743680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41241188 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138100100120576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124904096853395 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚177 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18393222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1820898215315 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138100076180480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123083198638080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123083181598720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28813188 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72778780 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31537592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41221126 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140019809176576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72758716 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919732996096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1671.579468332 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28813627 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1820243777359 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10419968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18393220 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41241188 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41221126 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31537590 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124903425376079 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128197863121408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826963941888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126370242804736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31537592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919732996096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1671.579468332 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28813627 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1820243777359 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10419968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18393220 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41241188 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41221126 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31537590 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124903425376079 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128197863121408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826963941888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126370242804736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31537592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18393222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1820898215315 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138100076180480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123083198638080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123083181598720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28813188 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72778780 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31537592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41221126 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140019809176576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_tr6üxãain_batch_end/active.all.freed‚72758716 Fï‚(:³ ÀÖãÔ´õë  disk.drbd1716.out‚0 disk.dm-1.out‚ 77.8046875 disk.drbd2265.in‚0 disk.drbd2080.in‚0* proc.memory.percent‚17.177820205688477* disk./.usagePercent‚15.912560462835406% disk./.usageGB‚237.10865020751953 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 77.8046875 disk.drbd2266.in‚0 disk.drbd2080.out‚0 network.sent‚ 4.199294e+06 network.recv‚655763 disk.rbd1.out‚ 9.259765625 disk.sda3.in‚0 disk.sda3.out‚ 77.8046875 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚3775.4052734375 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚8.34375# proc.memory.availableMB‚5.84375$ disk.nvme0n1.out‚3775.4052734375 disk.dm-0.in‚8.34375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2814.4140625 proc.cpu.threads‚17! disk.dm-0.out‚3775.4052734375 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0# memory_percent‚99.9643325805664 disk.nvme0n1.in‚8.34375 cpu‚25.23080147983137” ®…2ð- _wandbruntime‚1707  _runtime‚1707{À8ÇñÁ & lr-AdamW/pg2‚0.0002689054509299959 trainer/global_step‚2249 % _timestamp‚1.7884066041838503e+09 _runtime‚1707.171274781 _step‚179 & lr-AdamW/pg1‚0.0002689054509299959wã‹ÈÌòÆ trainer/global_step‚2249 % _timestamp‚1.7884066041838503e+09 _runtime‚1707.171274781 _step‚179 & lr-AdamW/pg1‚0.0002689054509299959 & lr-AdamW/pg2‚0.0002689054509299959³à}7Ï2ó- _wandbruntime‚1707  _runtime‚1707j šµØ'ôÒO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32224374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129131523705344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860262681803 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29440056 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130998654025216 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130998654025216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141120962326016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141120986266112 _runtime‚1707.172173977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1867130319872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867130319872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125773483996160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32224376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42148760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1961881292800 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42148760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129131523705344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130999328389632 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42128724 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129131541708288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74373136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129131541708288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125773501035520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143083024892416‹»”Ý'õ×O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42148760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1961881292800 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42148760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129131523705344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130999328389632 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42128724 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129131541708288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74373136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129131541708288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125773501035520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143083024892416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32224374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129131523705344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860262681803 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29440056 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130998654025216 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130998654025216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141120962326016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141120986266112 _runtime‚1707.172173977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1867130319872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867130319872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125773483996160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32224376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098´|äœ2ö- _wandbruntime‚1707  _runtime‚1707y~¬vC÷ÝM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚181 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10650700 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141182613063680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861733554723 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884066048554502e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32238390 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131056500869632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29452775 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚1707.843340094 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131055826491904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129187894747136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1868606122496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74385568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18801622 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18801620 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10651153 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42147178 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32238390 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143145365445120 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129187876744192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29452320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143145546705408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127690138720803 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74385568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129187876744192 trainer/global_step‚2249 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141182637003776 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42147178 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127689467243487 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1868606122496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32238392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1962909701632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Gg2óè&øâM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10650700 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141182613063680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861733554723 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884066048554502e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32238390 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131056500869632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29452775 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚1707.843340094 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131055826491904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129187894747136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1868606122496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74385568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18801622 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18801620 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10651153 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42147178 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32238390 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143145365445120 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129187876744192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29452320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143145546705408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127690138720803 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74385568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129187876744192 trainer/global_step‚2249 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141182637003776 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42147178 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127689467243487 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1868606122496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32238392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1962909701632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚181 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656µ™H«2ù- _wandbruntime‚1707  _runtime‚1707ÙóÎUVúÐ /loss/FRt1/detection_loss‚3.1267798476748996 /loss/FBt3/detection_loss‚0.7673266057993835 epoch‚1 /loss/FBt1/detection_loss‚3.4800810158252715 _step‚182 /loss/FRt2/assignment_loss‚4.766576744891979 2loss/SRqqt4/assignment_loss‚2.0482871018204034 % _timestamp‚1.7884066048570156e+09 2loss/SRqqt2/assignment_loss‚1.9076800964227536 2loss/SRqqt3/assignment_loss‚1.6361639677946056 .loss/FBt4/detection_loss‚4.150348365306854 .loss/FRt3/assignment_loss‚2.52690277894338 0loss/FBt3/assignment_loss‚0.2313858410785131 /loss/FRt4/detection_loss‚2.5627556052058935 0loss/SRqqt4/detection_loss‚1.966809019446373 trainer/global_step‚2249 /loss/FBt4/assignment_loss‚0.213519386947155 1loss/SRqqt3/detection_loss‚1.2796883451353227 /loss/FRt1/assignment_loss‚4.016937900472571 0loss/FBt1/assignment_loss‚0.4311973906007324 /loss/FRt3/detection_loss‚2.2651655034886464 0loss/SRqqt2/detection_loss‚2.118896378249657 /loss/FRt4/assignment_loss‚4.218348177770774 /loss/FRt2/detection_loss‚2.6511926319864063 1loss/SRqqt1/detection_loss‚2.5022155690193175 /loss/FBt2/detection_loss‚2.3113621612389883 %loss/total_loss‚54.29587383630114 _runtime‚1707.844364015 2loss/SRqqt1/assignment_loss‚2.2991275453567503 0loss/FBt2/assignment_loss‚0.8171258558245124ß–±k[ûÕ % _timestamp‚1.7884066048570156e+09 2loss/SRqqt2/assignment_loss‚1.9076800964227536 2loss/SRqqt3/assignment_loss‚1.6361639677946056 .loss/FBt4/detection_loss‚4.150348365306854 .loss/FRt3/assignment_loss‚2.52690277894338 0loss/FBt3/assignment_loss‚0.2313858410785131 /loss/FRt4/detection_loss‚2.5627556052058935 0loss/SRqqt4/detection_loss‚1.966809019446373 trainer/global_step‚2249 /loss/FBt4/assignment_loss‚0.213519386947155 1loss/SRqqt3/detection_loss‚1.2796883451353227 /loss/FRt1/assignment_loss‚4.016937900472571 0loss/FBt1/assignment_loss‚0.4311973906007324 /loss/FRt3/detection_loss‚2.2651655034886464 0loss/SRqqt2/detection_loss‚2.118896378249657 /loss/FRt4/assignment_loss‚4.218348177770774 /loss/FRt2/detection_loss‚2.6511926319864063 1loss/SRqqt1/detection_loss‚2.5022155690193175 /loss/FBt2/detection_loss‚2.3113621612389883 %loss/total_loss‚54.29587383630114 _runtime‚1707.844364015 2loss/SRqqt1/assignment_loss‚2.2991275453567503 0loss/FBt2/assignment_loss‚0.8171258558245124 /loss/FRt1/detection_loss‚3.1267798476748996 /loss/FBt3/detection_loss‚0.7673266057993835 epoch‚1 /loss/FBt1/detection_loss‚3.4800810158252715 _step‚182 /loss/FRt2/assignment_loss‚4.766576744891979 2loss/SRqqt4/assignment_loss‚2.0482871018204034¶›·Œ$‘ü‚(:† ÏÖãÔ¡¤Ÿ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.937 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦µ…?ý‚(:´ ÏÖãԧȔ  disk.rbd1.in‚0 disk.sda.out‚ 78.390625 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.177820205688477$ disk.nvme0n1.out‚3775.6357421875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0c0n1.out‚3775.6357421875* disk./.usagePercent‚15.912560462835406 disk.sda.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 network.sent‚ 4.205265e+06% disk./.usageGB‚237.10865020751953 disk.drbd1716.in‚0 disk.nvme0n1.in‚8.34375 disk.dm-1.out‚ 78.390625 disk.drbd2265.in‚0 disk.sda3.out‚ 78.390625 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚659753 disk.drbd1716.out‚0! disk.dm-0.out‚3775.6357421875 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚8.34375$ memory_percent‚99.96428489685059" proc.memory.rssMB‚ 2814.4140625 cpu‚25.229228755513216 disk.rbd1.out‚ 9.259765625 disk.dm-0.in‚8.34375 disk.dm-1.in‚0% proc.memory.availableMB‚ 5.85156250n‹§þ‚(:… ÞÖãÔª‡ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡aŽ¿Bÿ‚(:· ÞÖãÔÝÂÀ  disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0* proc.memory.percent‚17.177820205688477 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2814.4140625 disk.sda1.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚3776.03076171875 cpu‚25.2275694937322 disk.dm-1.out‚ 78.890625 disk.sda3.out‚ 78.890625 disk.rbd0.in‚0 disk.nvme0n1.in‚8.34375% disk.nvme0n1.out‚3776.03076171875 network.sent‚ 4.313283e+06% disk./.usageGB‚237.10871124267578 disk.rbd0.out‚0 disk.nvme0c0n1.in‚8.34375$ memory_percent‚99.96464252471924 network.recv‚665587 disk.sda.out‚ 78.890625 disk.rbd1.out‚ 9.4013671875" disk.dm-0.out‚3776.03076171875* disk./.usagePercent‚15.912564558955955 disk.drbd2265.in‚0 disk.drbd1716.in‚0& proc.memory.availableMB‚ 5.79296875 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.dm-0.in‚8.34375òú ’€‚(:‡ íÖãÔ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.777' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7554 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜O&C‚(:¸ íÖãÔÇî×  disk.drbd1716.in‚0 disk.dm-0.in‚8.34375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 network.sent‚ 4.319572e+06* disk./.usagePercent‚15.912560718842942 disk.rbd0.in‚0 disk.dm-1.out‚ 79.6328125 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚8.34375% disk.nvme0n1.out‚3776.75341796875 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚8.34375' disk.nvme0c0n1.out‚3776.75341796875 disk.rbd1.out‚ 9.4873046875 disk.drbd2265.in‚0" disk.dm-0.out‚3776.75341796875 disk.sda3.out‚ 79.6328125$ memory_percent‚99.96614456176758" proc.memory.rssMB‚ 2814.4140625* proc.memory.percent‚17.177820205688477$ disk./.usageGB‚237.1086540222168 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 79.6328125 network.recv‚670226 disk.sda2.in‚0 disk.drbd2265.out‚0$ proc.memory.availableMB‚5.546875 cpu‚25.22605501343369 proc.cpu.threads‚17 disk.sda1.in‚0 ‚°é2‚- _wandbruntime‚1743  _runtime‚1743#ÑóǃÁ & lr-AdamW/pg1‚0.0002683045422813879 & lr-AdamW/pg2‚0.0002683045422813879 trainer/global_step‚2299 % _timestamp‚1.7884066400941179e+09 _runtime‚1743.081564892 _step‚183ÙLOÐÌ„Æ % _timestamp‚1.7884066400941179e+09 _runtime‚1743.081564892 _step‚183 & lr-AdamW/pg1‚0.0002683045422813879 & lr-AdamW/pg2‚0.0002683045422813879 trainer/global_step‚2299·ñCÚó2…- _wandbruntime‚1743  _runtime‚1743DT‚;·&†ÓO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30076801 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30076361 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32925174 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133857966137856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1908116125696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19201854 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131949193650688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133857291773440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32925176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19201852 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43054776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908116125696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133857291773440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 _step‚184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131949193650688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10874509 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2005010566656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131949175647744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.sm§$A"all_pool.peak‚ 863814656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133857966137856Ù ˆÞ'‡ØO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133857966137856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1908116125696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19201854 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131949193650688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133857291773440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32925176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19201852 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43054776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908116125696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133857291773440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 _step‚184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131949193650688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10874509 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2005010566656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131949175647744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133857966137856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30076801 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30076361 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32925174 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024¸s·–2ˆ- _wandbruntime‚1743  _runtime‚1743´}Gá&‰ÛM $ _timestamp‚1.788406640778159e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚132005546689536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132005528686592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43073230 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1909591928320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43093292 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133915138617856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32939192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30089504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30089045 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚ 1743.76600242 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144265173886976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2005700208640 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32939190 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128573594654720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19210020 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130475509110895 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1901914456175 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133915138617856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19210022 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1909591928320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132005546689536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146271039804416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2299 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43093292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10879482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430þöï æ&ŠàM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30089045 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚ 1743.76600242 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144265173886976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2005700208640 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32939190 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128573594654720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19210020 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130475509110895 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1901914456175 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133915138617856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19210022 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1909591928320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132005546689536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146271039804416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2299 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43093292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10879482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 $ _timestamp‚1.788406640778159e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚132005546689536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132005528686592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43073230 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1909591928320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43093292 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133915138617856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32939192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30089504¹ÀÇ2‹- _wandbruntime‚1743  _runtime‚1743à›ÃÓVŒÐ /loss/FRt3/assignment_loss‚1.745926872739252 0loss/FRt4/assignment_loss‚3.8537702163060508 0loss/FBt2/assignment_loss‚0.7471874269346396 1loss/SRqqt2/detection_loss‚2.0598884224891663 trainer/global_step‚2299 2loss/SRqqt4/assignment_loss‚1.8246756776546438 .loss/FBt1/detection_loss‚3.757872521877289 _step‚186 /loss/FRt2/detection_loss‚2.2951469959484205 /loss/FRt4/detection_loss‚2.2415337984760604 1loss/SRqqt4/detection_loss‚1.8885643122096856 /loss/FRt3/detection_loss‚1.7914090422245692 1loss/SRqqt1/assignment_loss‚2.424266193062067 2loss/SRqqt2/assignment_loss‚1.8653974950313568 1loss/FBt1/assignment_loss‚0.32159520932812785 /loss/FBt4/assignment_loss‚2.520824432373047 $ _timestamp‚1.788406640780111e+09 _runtime‚1743.767483795 /loss/FRt1/assignment_loss‚4.107210099697113 2loss/SRqqt3/assignment_loss‚1.5736610970959284 1loss/FBt3/assignment_loss‚0.23673487894923947 1loss/SRqqt3/detection_loss‚1.1456944406896397 .loss/FBt2/detection_loss‚2.299326298137506 .loss/FBt4/detection_loss‚8.443632781505585 epoch‚1 /loss/FRt2/assignment_loss‚4.203998542494244 /loss/FBt3/detection_loss‚0.5556727595096176 .loss/FRt1/detection_loss‚2.686075293355518 0loss/SRqqt1/detection_loss‚2.413382899016142 %loss/total_loss‚57.00344770710491['d–GÕ .loss/FRt1/detection_loss‚2.686075293355518 0loss/SRqqt1/detection_loss‚2.413382899016142 %loss/total_loss‚57.00344770710491 /loss/FRt3/assignment_loss‚1.745926872739252 0loss/FRt4/assignment_loss‚3.8537702163060508 0loss/FBt2/assignment_loss‚0.7471874269346396 1loss/SRqqt2/detection_loss‚2.0598884224891663 trainer/global_step‚2299 2loss/SRqqt4/assignment_loss‚1.8246756776546438 .loss/FBt1/detection_loss‚3.757872521877289 _step‚186 /loss/FRt2/detection_loss‚2.2951469959484205 /loss/FRt4/detection_loss‚2.2415337984760604 1loss/SRqqt4/detection_loss‚1.8885643122096856 /loss/FRt3/detection_loss‚1.7914090422245692 1loss/SRqqt1/assignment_loss‚2.424266193062067 2loss/SRqqt2/assignment_loss‚1.8653974950313568 1loss/FBt1/assignment_loss‚0.3215952093281278 È•5 /loss/FBt4/assignment_loss‚2.520824432373047 $ _timestamp‚1.788406640780111e+09 _runtime‚1743.767483795 /loss/FRt1/assignment_loss‚4.107210099697113 2loss/SRqqt3/assignment_loss‚1.5736610970959284 1loss/FBt3/assignment_loss‚0.23673487894923947 1loss/SRqqt3/detection_loss‚1.1456944406896397 .loss/FBt2/detection_loss‚2.299326298137506 .loss/FBt4/detection_loss‚8.443632781505585 epoch‚1 /loss/FRt2/assignment_loss‚4.203998542494244 /loss/FBt3/detection_loss‚0.5556727595096176ºš§I’Ž‚(:‡ üÖãÔÚŽ¨ gpu.0.gpu‚43.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D BD‚(:¹ üÖãÔñ´™  disk.drbd2080.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 9.43359375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2814.4140625 network.recv‚673897" disk.dm-0.out‚3779.42431640625 disk.drbd2266.out‚0 disk.sda1.in‚0 cpu‚25.224415748846752 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚8.34375' disk.nvme0c0n1.out‚3779.42431640625$ disk./.usageGB‚234.4545440673828 disk.dm-0.in‚8.34375 disk.dm-1.out‚ 80.9140625 disk.drbd2266.in‚0 disk.rbd0.in‚0 network.sent‚ 4.3248e+06 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 80.9140625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚99.94242191314697* proc.memory.percent‚17.177820205688477 disk.drbd2080.out‚0* disk./.usagePercent‚15.734441172827452 disk.rbd1.out‚ 9.6357421875 disk.nvme0n1.in‚8.34375% disk.nvme0n1.out‚3779.42431640625 disk.sda.in‚0 disk.sda.out‚ 80.9140625 disk.sda3.in‚0Š9bª’‚(:‡ ‹×ãÔ¦’• gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.679' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3358 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Є—¾I‘‚(:¾ ‹×ãÔªŽÄ  network.sent‚ 4.433229e+06 disk.nvme0c0n1.in‚8.34375' disk.nvme0c0n1.out‚3779.76416015625 disk.drbd1716.in‚0 disk.sda.in‚0 disk.sda.out‚ 81.45703125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚8.34375% disk.nvme0n1.out‚3779.76416015625 disk.drbd2265.out‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 9.54296875" proc.memory.rssMB‚ 2814.4140625* proc.memory.percent‚17.177820205688477 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 cpu‚25.223117148949104 network.recv‚680148* disk./.usagePercent‚15.734441172827452 disk.dm-1.in‚0 disk.dm-1.out‚ 81.45703125 disk.drbd2265.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚234.4545440673828 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 9.6357421875 disk.dm-0.in‚8.34375 disk.sda3.out‚ 81.45703125$ memory_percent‚99.94175434112549 disk.sda2.out‚0" disk.dm-0.out‚3779.76416015625 disk.rbd0.in‚0wÆü–2’- _wandbruntime‚1779  _runtime‚17798¸–ØÇ“Á & lr-AdamW/pg1‚0.0002676763803151696 & lr-AdamW/pg2‚0.0002676763803151696 trainer/global_step‚2349 % _timestamp‚1.7884066761905756e+09 _runtime‚1779.178030855 _step‚187è%_Ì”Æ _runtime‚1779.178030855 _step‚187 & lr-AdamW/pg1‚0.0002676763803151696 & lr-AdamW/pg2‚0.0002676763803151696 trainer/global_step‚2349 % _timestamp‚1.7884066761905756e+09»Œ–Œ2•- _wandbruntime‚1779  _runtime‚1779°£õÕØ'–ÒO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134766845593088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136715929521664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19610254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚188 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77606802 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19610252 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2047971559424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1949101931520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30714878 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1942587798367 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44000864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44000864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1949758292992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149333833541120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43980828 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1949758292992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11104175 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147286060032512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77626840 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _runtime‚ 1779.17896075 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30714427 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149334031591936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚2349 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11104624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174110720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1941933360619 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136716603886080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133205830412779 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33625974 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43980828 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136715929521664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131263914091520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131263897052160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33625974 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43980828 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136715929521664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131263914091520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131263897052160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136715929521664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19610254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚188 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77606802 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19610252 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2047971559424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1949101931520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30714878 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1942587798367 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416¼óÚò2˜- _wandbruntime‚1779  _runtime‚1779~D“] ™ÝM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134823198631936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43999282 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133262222455619 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131318801182720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1943404233539 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884066768664103e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136773776366080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33639992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136773101988352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1950577734144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11109152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77639272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33639990 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30727125 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11108705 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33639990 trainer/global_step‚2349 _runtime‚1779.854454168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30727574 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33639992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1942749795583 ^JDeviceSIVã×átatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134823198631936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19618420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147347710770176 _step‚189 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133261550978303 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134823180628992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131318818222080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44019344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1949921359360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149396532016640ïIuIè&šâM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44019344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136773776366080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1950577734144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77659336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134823180628992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136773101988352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1950577734144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11109152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77639272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33639990 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30727125 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11108705 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33639990 trainer/global_step‚2349 _runtime‚1779.854454168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30727574 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33639992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1942749795583 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134823198631936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19618420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147347710770176 _step‚189 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133261550978303 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134823180628992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131318818222080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44019344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1949921359360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149396532016640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134823198631936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43999282 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133262222455619 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131318801182720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1943404233539 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884066768664103e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136773776366080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33639992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34326774 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139575241634304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2091122470912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44906880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152459529724416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139574567269888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2090956748800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134009120619520 _runtime‚1815.162330818 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1990744098816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20018654 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34326774 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134009103580160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1990087737344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34326776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44926916 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79233654 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139575241634304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚2399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1990087737344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137584497535488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1982768700027 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844Îs¨Ùš ©ØO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 % _timestamp‚1.7884067121746984e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139574567269888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2090956748800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134009120619520 _runtime‚1815.162330818 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1990744098816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20018654 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34326774 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134009103580160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1990087737344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34326776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44926916 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79233654 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139575241634304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚2399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1990087737344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137584497535488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1982768700027 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44906880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137584479532544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1990744098816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34326774 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139575241634304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2091122470912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44906880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152459529724416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1990907165184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20026822 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152522041733632 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139632414114304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34340792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137640832571392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134064007710720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31365623 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44945396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136048264323027 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150430223713280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884067128491435e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44925334 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136047592845711 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1984239572947 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚2399 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79286188 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31366077 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44925334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11339255 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134064024750080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20026820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11338803 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34340790 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚193 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1991563539968ØÌ:‘è&¬âM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚1815.837470002 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44945396 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150430223713280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884067128491435e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44925334 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136047592845711 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1984239572947 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚2399 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79286188 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31366077 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44925334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11339255 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134064024750080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20026820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11338803 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34340790 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚193 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1991563539968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34340792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1990907165184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1983585134991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79266124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1991563539968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139631739736576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137640850574336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1990907165184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20026822 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152522041733632 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139632414114304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34340792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137640832571392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134064007710720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31365623 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44945396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136048264323027 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863814656 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚136781693315072 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚142484098320896 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚11642226 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2125588519936 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚140459858681856 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚45635210 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚155521323205120 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953744 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚142483587488768 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829184 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚11642078 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 $ _timestamp‚1.788406773583271e+09 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚35122572 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585024 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚140459876684800 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2023728806912 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚136781676275712 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2016279805479 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 _runtime‚1876.571066764 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2024221636096 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863814656 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚136781693315072 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚142484098320896 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚11642226 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2125588519936 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚140459858681856 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚45635210 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚155521323205120 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953744 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚142483587488768 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829184 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚11642078 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 $ _timestamp‚1.788406773583271e+09 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚35122572 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585024 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚140459876684800 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2023728806912 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚136781676275712 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2016279805479 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 _runtime‚1876.571066764 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2024221636096 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚153433747606528 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚142521293476352 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137187840 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2016643414887 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2125953646592 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32145914 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚136816987074560 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚136816970035200 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 _step‚196 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚80775892 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚11646720 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2024586762752 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2024093933568 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953744 % _timestamp‚1.7884067745496683e+09 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚35133380 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2125840398848 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚80759915 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚142520782644224 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚148 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24399 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚155559701253120 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15975 BDeviceSt½lÅ[ atsMonitor.on_validation_batch_end/segment.all.current‚844 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2024093933568 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚20499046 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829184 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚35133380 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚138834121403831 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35133378 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚45626537 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚140496706713600 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829184 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚45642512 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863814656 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚153433723666432 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914384 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚35133378 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚153433747606528 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚142521293476352 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137187840 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2016643414887 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2125953646592 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32145914 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚136816987074560 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚136816970035200 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 _step‚196 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚80775892 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚11646720 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2024586762752 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2024093933568 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953744 % _timestamp‚1.7884067745496683e+09 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚35133380 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2125840398848 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚80759915 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚142520782644224 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚148 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24399 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚155559701253120 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15975 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2024093933568 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚20499046 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829184 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚35133380 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚138834121403831 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35133378 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚45626537 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚140496706713600 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829184 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚45642512 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863814656 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚153433723666432 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914384 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚35133378 B&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.32027289271354675 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09999565780162811 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3456100821495056 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.38613325357437134 jet/accuracy_4_of_5‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 )jet/accuracy_1_of_6‚0.942307710647583 jet/accuracy_2_of_7‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9899783134460449 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.992368757724762 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4316903352737427 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.40326279401779175 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.42973166704177856 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11073404550552368 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14583294093608856 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3826797306537628 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.42556217312812805 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3801362216472626 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0631052628159523 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0068755862303078175 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 )loss/val_total_loss‚51.45730432571739 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9726063013076782 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3685924708843231 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 )jet/accuracy_2_of_6‚0.557692289352417 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333730697632 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.984702467918396 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4171442687511444 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909341931343079 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.39994871616363525 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15350037813186646 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.12121212482452393 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_4‚1 ,"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.375 $ _timestamp‚1.788406787313307e+09 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.41660183668136597 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 epoch‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4027027487754822 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_3_of_3‚0.017449481412768364 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.42930886149406433 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17513252794742584 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0873015895485878 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5069764852523804 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0042087542824447155 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2638393044471741 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333730697632 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.810529887676239 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2879602611064911 +particle/specificity‚0.9436262845993042 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9919822216033936 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3149224817752838 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.37934693694114685 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009145880118012428 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.088048554956913 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11261261999607086 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5215517282485962 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23273353278636932 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4225691854953766 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_5‚1 jet/accuracy_4_of_6‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3833160698413849 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43599262833595276 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16493597626686096 particle/accuracy_2_of_6‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3454657793045044 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.07692307978868484 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚2417 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.38063061237335205 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21171101927757263 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3854750096797943 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4144429564476013 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4301217198371887 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22471816837787628 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9984127283096313 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.36561426520347595 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt4SRqqt0FBt/FRt_purity‚T÷PÃ×M0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9779828190803528 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_6‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 jet/accuracy_6_of_7‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.43301284313201904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07961279898881912 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1942676305770874 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 6validation_average_jet_accuracy‚0.3856169283390045 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 particle/accuracy_5_of_7‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.30153733491897583 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3650839924812317 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.35654547810554504 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6576427221298218 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3047178089618683 particle/accuracy_5_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _runtime‚1890.302993328 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03656454384326935 jet/accuracy_1_of_7‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9666666984558105 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4061129689216614 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7870593070983887 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09314720332622528 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02065434493124485 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.42120400071144104 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10789556801319122 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4861111044883728 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12165039777755737 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.6773557662963867 ,jet/accuracy_3_of_4‚0.026595400646328926 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3331720232963562 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4470270574092865 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.09883040189743042 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29124006628990173 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4075896441936493 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9867526292800903 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4528742730617523 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08871322125196457 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9971379041671753 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14465796947479248 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9931172132492065 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9942470788955688 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 particle/accuracy_1_of_7‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4944915473461151 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08857662230730057 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0838896855711937 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2023523896932602 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.026529645547270775 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11752873659133911 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3475877344608307 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9951983094215393 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08920153230428696 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.43174412846565247 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0020101526752114296 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.28893500566482544 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1466602385044098 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13201618194580078 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9818710088729858 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9847960472106934 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11472978442907333 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3333333730697632 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.05999999865889549 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9866707921028137 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14166076481342316 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.001873897621408105 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2813389301300049 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_1_of_5‚1 ) Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3580191135406494 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.38319215178489685 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1636904776096344 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.39227569103240967 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09142415970563889 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1527777761220932 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010245713405311108 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08906441926956177 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12117299437522888 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9891591668128967 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04885057732462883 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196217060089111 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9872637391090393 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.38075387477874756 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6231651902198792 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11145977675914764 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.38627445697784424 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9880948066711426 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.015476190485060215 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4680481553077698 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2907407879829407 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07951481640338898 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.023339074105024338 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3821548819541931 jet/accuracy_7_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2557218670845032 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33645832538604736 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06669876724481583 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4367910623550415 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09093905985355377 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 particle/accuracy_6_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3991791307926178 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 particle/accuracy_2_of_2‚1 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.056737590581178665 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2178160846233368 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32117128372192383 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4052083492279053 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.14176426827907562 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4087506830692291 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4289821982383728 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04887862130999565 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2766276001930237 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3913690149784088 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 particle/accuracy_4_of_7‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3554598093032837 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3993055522441864 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3827454149723053 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06818182021379471 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.44295257329940796 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_1_of_6‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09059096872806549 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9971379041671753 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4646950364112854 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9403735399246216 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.25757578015327454 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935297012329 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9949214458465576 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.43287035822868347 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14983442425727844 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14212505519390106 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05397070199251175 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0¦V.ô2Âþ¥ 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3331720232963562 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4470270574092865 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.09883040189743042 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29124006628990173 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4075896441936493 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9867526292800903 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4528742730617523 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08871322125196457 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9971379041671753 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14465796947479248 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9931172132492065 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9942470788955688 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 particle/accuracy_1_of_7‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4944915473461151 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08857662230730057 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0838896855711937 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2023523896932602 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.026529645547270775 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11752873659133911 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3475877344608307 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9951983094215393 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08920153230428696 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.43174412846565247 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0020101526752114296 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.28893500566482544 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1466602385044098 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13201618194580078 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9818710088729858 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9847960472106934 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11472978442907333 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3333333730697632 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.05999999865889549 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9866707921028137 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14166076481342316 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.001873897621408105 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2813389301300049 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_1_of_5‚1 ) Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3580191135406494 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.38319215178489685 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1636904776096344 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.39227569103240967 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09142415970563889 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1527777761220932 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010245713405311108 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08906441926956177 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12117299437522888 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9891591668128967 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04885057732462883 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196217060089111 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9872637391090393 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.38075387477874756 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6231651902198792 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11145977675914764 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.38627445697784424 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9880948066711426 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.015476190485060215 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4680481553077698 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2907407879829407 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07951481640338898 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.023339074105024338 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3821548819541931 jet/accuracy_7_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2557218670845032 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33645832538604736 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06669876724481583 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4367910623550415 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09093905985355377 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 particle/accuracy_6_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3991791307926178 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 particle/accuracy_2_of_2‚1 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.056737590581178665 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2178160846233368 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32117128372192383 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4052083492279053 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.14176426827907562 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4087506830692291 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4289821982383728 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04887862130999565 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2766276001930237 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.410900294780`.ƒê`7312 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3913690149784088 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 particle/accuracy_4_of_7‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3554598093032837 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3993055522441864 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3827454149723053 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06818182021379471 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.44295257329940796 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_1_of_6‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09059096872806549 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9971379041671753 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4646950364112854 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9403735399246216 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.25757578015327454 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935297012329 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9949214458465576 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.43287035822868347 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14983442425727844 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14212505519390106 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05397070199251175 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07271359860897064 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9744047522544861 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1962963044643402 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008660871535539627 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.016981132328510284 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10379424691200256 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.0795019194483757 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.18257877230644226 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.39181753993034363 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9825900197029114 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34136906266212463 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997843861579895 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8418951630592346 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_2_of_3‚0.19340382516384125 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16406570374965668 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05542238429188728 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.07624807953834534 particle/accuracy_3_of_4‚1 jet/accuracy_6_of_6‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.08801005035638809 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07961279898881912 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666666626930237 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3004601299762726 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.39828863739967346 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_4‚0.2935284972190857 (particle/accuracy‚0.8412543535232544 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9783638119697571 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3769841492176056 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.07098764926195145 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.47942912578582764 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027956990525126457 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.42973166704177856 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14516954123973846 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.40455445647239685 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08164303749799728 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3924591541290283 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.27136948704719543 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9318612813949585 _step‚197 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.39382901787757874 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.32375475764274597 (particle/f_score‚0.36144641041755676 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9568932056427002 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9762265086174011 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.32027289271354675 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09999565780162811 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3456100821495056 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.38613325357437134 jet/accuracy_4_of_5‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 )jet/accuracy_1_of_6‚0.942307710647583 jet/accuracy_2_of_7‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9899783134460449 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.992368757724762 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4316903352737427 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.40326279401779175 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.42973166704177856 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11073404550552368 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14583294093608856 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3826797306537628 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.42556217312812805 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3801362216472626 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0631052628159523 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0068755862303078175 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 )loss/val_total_loss‚51.45730432571739 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9726063013076782 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3685924708843231 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 )jet/accuracy_2_of_6‚0.557692289352417 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333730697632 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.984702467918396 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4171442687511444 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909341931343079 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.39994871616363525 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15350037813186646 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.12121212482452393 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_4‚1 ,"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.375 $ _timestamp‚1.788406787313307e+09 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.41660183668136597 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 epoch‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4027027487754822 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_3_of_3‚0.017449481412768364 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.42930886149406433 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17513252794742584 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0873015895485878 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5069764852523804 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0042087542824447155 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2638393044471741 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333730697632 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.810529887676239 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2879602611064911 +particle/specificity‚0.9436262845993042 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9919822216033936 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3149224817752838 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.37934693694114685 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009145880118012428 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.088048554956913 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11261261999607086 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5215517282485962 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23273353278636932 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4225691854953766 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_5‚1 jet/accuracy_4_of_6‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3833160698413849 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43599262833595276 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16493597626686096 particle/accuracy_2_of_6‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3454657793045044 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.07692307978868484 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚2417 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.38063061237335205 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21171101927757263 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3854750096797943 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4144429564476013 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4301217198371887 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22471816837787628 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9984127283096313 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.36561426520347595 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9779828190803528 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_6‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 jet/accuracy_6_of_7‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.43301284313201904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07961279898881912 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1942676305770874 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 6validation_average_jet_accuracy‚0.3856169283390045 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 particle/accuracy_5_of_7‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.30153733491897583 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3650839924812317 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.35654547810554504 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6576427221298218 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3047178089618683 particle/accuracy_5_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _runtime‚1890.302993328 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03656454384326935 jet/accuracy_1_of_7‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9666666984558105 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4061129689216614 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7870593070983887 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09314720332622528 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02065434493124485 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.42120400071144104 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10789556801319122 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4861111044883728 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12165039777755737 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.6773557662963867 ,jet/accuracy_3_of_4‚0.026595400646328926Å•(ü˒Â(:‡ ’ØãÔèþ– gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)UîâIÄ‚(:¾ ’ØãÔ†œ¸  disk.dm-1.out‚ 86.19921875! proc.memory.rssMB‚ 2820.703125* proc.memory.percent‚17.216205596923828 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd2265.out‚0 disk.sda.out‚ 86.19921875' proc.memory.availableMB‚ 951.3671875 proc.cpu.threads‚17 disk.sda1.out‚0% disk.nvme0n1.out‚4260.41064453125" disk.dm-0.out‚4260.41064453125 disk.drbd2265.in‚0 disk.sda3.out‚ 86.19921875 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda.in‚0* disk./.usagePercent‚15.765599337804767 disk.sda2.in‚0 disk.drbd1716.out‚0$ memory_percent‚94.19331550598145 cpu‚25.36755985697523$ disk./.usageGB‚234.9188232421875 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 10.2705078125 disk.sda3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375 network.sent‚ 4.793032e+06 network.recv‚724197' disk.nvme0c0n1.out‚4260.41064453125 disk.dm-0.in‚8.34375 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚0l5N„2Å- _wandbruntime‚1917  _runtime‚1917yÂÇÆÁ & lr-AdamW/pg1‚0.0002663388257142208 & lr-AdamW/pg2‚0.0002663388257142208 trainer/global_step‚2449 % _timestamp‚1.7884068143760507e+09 _runtime‚1917.363470507 _step‚198¡ƒPÌÇÆ trainer/global_step‚2449 % _timestamp‚1.7884068143760507e+09 _runtime‚1917.363470507 _step‚198 & lr-AdamW/pg1‚0.0002663388257142208 & lr-AdamW/pg2‚0.0002663388257142208ÆîÁú2È- _wandbruntime‚1917  _runtime‚1917?ó ÉÓO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157994536023552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚1917.364534394 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144777191164928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35708354 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11845200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142722423289344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144776516800000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2054093510656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20_‘Ÿ'Í039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157994729879552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2157424869888 _step‚199 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142722423289344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32681580 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46314927 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46294890 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82023281 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144777191164928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144776516800000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142722423289344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32681580 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46314927 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46294890 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82023281 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144777191164928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144776516800000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚1917.364534394 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144777191164928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35708354 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11845200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142722423289344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144776516800000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2054093510656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157994729879552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2157424869888 _step‚199 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20844974 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788406815048128e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144834363644928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158057038885376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2054912938496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722370 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139032721086464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11849239 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141080066060937 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46333407 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2047344974473 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32694211 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142778794331136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11849668 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144834363644928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82035712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2055569313792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46313344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2047999412433 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35722368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142778794331136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158057232728064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20844972 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82035712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46333407 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155898955747328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2054912938496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46313344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063{ÙN ÍáM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46333407 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155898955747328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2054912938496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46313344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20844974 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788406815048128e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144834363644928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158057038885376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2054912938496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722370 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139032721086464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11849239 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141080066060937 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46333407 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2047344974473 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32694211 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142778794331136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11849668 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144834363644928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82035712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2055569313792 SEDeviceStatsMonitor.on_train_batch_end/allocate©Sø•Éd_bytes.small_pool.peak‚ 863815168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46313344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2047999412433 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35722368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142778794331136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158057232728064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20844972 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82035712ȶBƒ2Î- _wandbruntime‚1918  _runtime‚1918郙VZÏÔ /loss/FBt2/assignment_loss‚0.501950167119503 0loss/FBt4/assignment_loss‚0.7660095451865345 % _timestamp‚1.7884068150501206e+09 epoch‚2 /loss/FRt3/assignment_loss‚2.023155594771763 /loss/FRt4/assignment_loss‚2.855942936738332 /loss/FRt2/detection_loss‚2.4312394872689858 1loss/SRqqt4/detection_loss‚1.4112683246275046 %loss/total_loss‚49.39002592466434 2loss/SRqqt1/assignment_loss‚2.3928651103266962 1loss/FBt1/assignment_loss‚0.16106932769266027 /loss/FRt4/detection_loss‚2.1879200014803146 _runtime‚1918.037500679 2loss/SRqqt2/assignment_loss‚2.2295961011858547 2loss/SRqqt3/assignment_loss‚1.3952708254066797 /loss/FRt1/detection_loss‚2.9425091181482586 /loss/FBt3/detection_loss‚0.8460685369327489 /loss/FRt1/assignment_loss‚3.424078106880188 .loss/FRt2/assignment_loss‚4.63755559309935 /loss/FBt4/detection_loss‚3.0447728782892227 _step‚201 1loss/FBt3/assignment_loss‚0.23973902228009225 1loss/SRqqt1/detection_loss‚2.7593669913433216 /loss/FBt1/detection_loss‚3.7092337608337402 2loss/SRqqt4/assignment_loss‚1.7944891743071667 .loss/FRt3/detection_loss‚1.773160651255891 1loss/SRqqt2/detection_loss‚1.7988797005485087 /loss/FBt2/detection_loss‚2.7616904377937317 trainer/global_step‚2449 1loss/SRqqt3/detection_loss‚1.3021945311472967X"ä_ÐÙ %loss/total_loss‚49.39002592466434 2loss/SRqqt1/assignment_loss‚2.3928651103266962 1loss/FBt1/assignment_loss‚0.16106932769266027 /loss/FRt4/detection_loss‚2.1879200014803146 _runtime‚1918.037500679 2loss/SRqqt2/assignment_loss‚2.2295961011858547 2loss/SRqqt3/assignment_loss‚1.3952708254066797 /loss/FRt1/detection_loss‚2.9425091181482586 /loss/FBt3/detection_loss‚0.8460685369327489 /loss/FRt1/assignment_loss‚3.424078106880188 .loss/FRt2/assignment_loss‚4.63755559309935 /loss/FBt4/detection_loss‚3.0447728782892227 _step‚201 1loss/FBt3/assignment_loss‚0.23973902228009225 1loss/SRqqt1/detection_loss‚2.7593669913433216 /loss/FBt1/detection_loss‚3.7092337608337402 2loss/SRqqt4/assignment_loss‚1.7944891743071667 .loss/FRt3/detection_loss‚1.773160651255891 1loss/SRqqt2/detection_loss‚1.7988797005485087 /loss/FBt2/detection_loss‚2.7616904377937317 trainer/global_step‚2449 1loss/SRqqt3/detection_loss‚1.3021945311472967 /loss/FBt2/assignment_loss‚0.501950167119503 0loss/FBt4/assignment_loss‚0.7660095451865345 % _timestamp‚1.7884068150501206e+09 epoch‚2 /loss/FRt3/assignment_loss‚2.023155594771763 /loss/FRt4/assignment_loss‚2.855942936738332 /loss/FRt2/detection_loss‚2.4312394872689858 1loss/SRqqt4/detection_loss‚1.4112683246275046Éôï‘æÑ‚(:… ¡ØãÔœ¸¥ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆïàKÒ‚(:À ¡ØãÔ꣩  disk.dm-1.out‚ 86.74609375 disk.drbd2265.in‚0 disk.sda.out‚ 86.74609375 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd1716.out‚0 disk.nvme0n1.in‚8.34375 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 cpu‚25.365114488224513' disk.nvme0c0n1.out‚4284.96337890625! proc.memory.rssMB‚ 2820.703125% disk./.usageGB‚234.91700744628906 disk.dm-0.in‚8.34375 disk.sda.in‚0 disk.sda3.out‚ 86.74609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚15.76547747821847" disk.dm-0.out‚4284.96337890625 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.2705078125% disk.nvme0n1.out‚4284.96337890625 disk.drbd1716.in‚0$ memory_percent‚94.19190883636475( proc.memory.availableMB‚ 951.59765625* proc.memory.percent‚17.216205596923828 proc.cpu.threads‚17 network.recv‚730911 network.sent‚ 4.917291e+06×éö‘Ó‚(:† °ØãÔÁ̧ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.975' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YuÜÚ;Ô‚(:° °ØãÔ›‡©  disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0* disk./.usagePercent‚15.765474662135592% disk./.usageGB‚234.91696548461914 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0& proc.memory.availableMB‚ 951.296875* proc.memory.percent‚17.216205596923828 cpu‚25.362729293548714 disk.rbd1.out‚ 10.3330078125 disk.dm-0.out‚4735.138671875 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚8.34375 disk.dm-1.out‚87.34375 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.out‚87.34375 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0% disk.nvme0c0n1.out‚4735.138671875 network.sent‚ 5.025014e+06 disk.sda.out‚87.34375$ memory_percent‚94.19374465942383! proc.memory.rssMB‚ 2820.703125 network.recv‚736402 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375 proc.cpu.threads‚17# disk.nvme0n1.out‚4735.138671875 disk.dm-0.in‚8.34375ßÔ\’Õ‚(:‡ ¿ØãÔ™Áÿ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"n´ÃCÖ‚(:¸ ¿ØãÔÆ¯Ñ  disk.drbd2265.in‚0 disk.sda.out‚ 87.91796875$ memory_percent‚94.19083595275879* disk./.usagePercent‚15.765469797992443 disk.drbd2265.out‚0 disk.sda.in‚0 network.sent‚ 5.03011e+06$ disk./.usageGB‚234.9168930053711 disk.rbd0.in‚0% disk.nvme0c0n1.out‚4735.435546875 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚4735.435546875' proc.memory.availableMB‚ 951.7734375! proc.memory.rssMB‚ 2820.703125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚17.216205596923828 network.recv‚739960 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375 disk.drbd1716.out‚0 disk.dm-0.in‚8.34375 disk.dm-0.out‚4735.435546875 cpu‚25.360245996796362 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.dm-1.out‚ 87.91796875 disk.sda3.out‚ 87.91796875 disk.rbd0.out‚0 disk.nvme0n1.in‚8.34375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.3330078125 disk.dm-1.in‚0‰_%æ2×- _wandbruntime‚1953  _runtime‚1953³RxÉØà _runtime‚1953.341400246 _step‚202 ' lr-AdamW/pg1‚0.00026562971182386437 ' lr-AdamW/pg2‚0.00026562971182386437 trainer/global_step‚2499 % _timestamp‚1.7884068503539128e+097²fÎÙÈ trainer/global_step‚2499 % _timestamp‚1.7884068503539128e+09 _runtime‚1953.341400246 _step‚202 ' lr-AdamW/pg1‚0.00026562971182386437 ' lr-AdamW/pg2‚0.00026562971182386437Ê «i˜2Ú- _wandbruntime‚1953  _runtime‚1953“ËÚ™Ù'ÛÓO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2095735678464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143810387362821 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2087363878917 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36409152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145540075231744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161120140085760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147635828913152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47240979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141723023483904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33316192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21245206 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143811058839933 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145540093234688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145540093234688 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2200132471296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36409154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2095079316480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147635154548224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12070532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161119950424064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147635154548224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47220942 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141723040523264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158919841892864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200298192896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47220942 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145540075231744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚1953.342355408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147635828913152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2095079316480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33315736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚203 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2200132471296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36409154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2095079316480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147635154548224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12070532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161119950424064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147635154548224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47220942 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141723040523264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158919841892864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200298192896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47220942 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145540075231744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚1953.342355408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147635828913152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2095079316480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33315736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚203 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚1954.022863284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147693001393152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143866107928345 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145596446273536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bythÙíues.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚204 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161182647244288 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83682629 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145596428270592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2096555119616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2088180313881 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33328956 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 trainer/global_step‚2499 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2095898744320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2095898744320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161182453401600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2088834751841 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145596446273536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47239396 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21253372 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36423168 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36423170 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2200984711168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884068510346305e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147693001393152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33328496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36423170 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2200984711168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884068510346305e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147693001393152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33328496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚1954.022863284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147693001393152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143866107928345 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145596446273536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚204 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161182647244288 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83682629 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145596428270592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2096555119616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2088180313881 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33328956 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 trainer/global_step‚2499 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2095898744320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2095898744320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161182453401600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2088834751841 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145596446273536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47239396 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21253372 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36423168 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844Ì„ÏÞñ2à- _wandbruntime‚1954  _runtime‚1954‘" SáÍ 0loss/FBt4/assignment_loss‚0.6464323202684076 /loss/FRt1/assignment_loss‚4.034493559115642 1loss/SRqqt1/assignment_loss‚2.239972472955019 0loss/FBt1/assignment_loss‚0.4263965375703833 1loss/SRqqt3/detection_loss‚1.1884883233006038 trainer/global_step‚2499 _step‚205 .loss/FRt2/assignment_loss‚3.96041215288228 0loss/FBt2/assignment_loss‚0.6629705939974103 1loss/SRqqt4/detection_loss‚1.5723703000389162 /loss/FRt4/detection_loss‚2.3816902313543404 _runtime‚1954.024203712 1loss/SRqqt4/assignment_loss‚2.081211829038917 -loss/FRt2/detection_loss‚2.04704975568015 1loss/SRqqt1/detection_loss‚2.2778008671907277 /loss/FBt2/detection_loss‚2.9303834097726003 /loss/FBt3/detection_loss‚0.6526070574525361 $ _timestamp‚1.788406851036833e+09 &loss/total_loss‚49.198214748009605 epoch‚2 0loss/FRt4/assignment_loss‚2.9151408413182134 1loss/SRqqt3/assignment_loss‚1.521422005480244 0loss/FBt3/assignment_loss‚0.2010364481130385 /loss/SRqqt2/detection_loss‚1.99454027137091 /loss/FBt1/detection_loss‚3.1080017052590847 /loss/FBt4/detection_loss‚3.4731585482756295 /loss/FRt3/assignment_loss‚2.271983085108585 .loss/FRt1/detection_loss‚2.840670804719667 /loss/FRt3/detection_loss‚1.9825916922971851 2loss/SRqqt2/assignment_loss‚1.7873899354491123{¯TñXâÒ /loss/FRt4/detection_loss‚2.3816902313543404 _runtime‚1954.024203712 1loss/SRqqt4/assignment_loss‚2.081211829038917 -loss/FRt2/detection_loss‚2.04704975568015 1loss/SRqqt1/detection_loss‚2.2778008671907277 /loss/FBt2/detection_loss‚2.9303834097726003 /loss/FBt3/detection_loss‚0.6526070574525361 $ _timestamp‚1.788406851036833e+09 &loss/total_loss‚49.198214748009605 epoch‚2 0loss/FRt4/assignment_loss‚2.9151408413182134 1loss/SRqqt3/assignment_loss‚1.521422005480244 0loss/FBt3/assignment_loss‚0.2010364481130385 /loss/SRqqt2/detection_loss‚1.99454027137091 /loss/FBt1/detection_loss‚3.1080017052590847 /loss/FBt4/detection_loss‚3.4731585482756295 /loss/FRt3/assignment_loss‚2.271983085108585 .loss/FRt1/detection_loss‚2.840670804719667 /loss/FRt3/detection_loss‚1.9825916922971851 2loss/SRqqt2/assignment_loss‚1.7873899354491123 0loss/FBt4/assignment_loss‚0.6464323202684076 /loss/FRt1/assignment_loss‚4.034493559115642 1loss/SRqqt1/assignment_loss‚2.239972472955019 0loss/FBt1/assignment_loss‚0.4263965375703833 1loss/SRqqt3/detection_loss‚1.1884883233006038 trainer/global_step‚2499 _step‚205 .loss/FRt2/assignment_loss‚3.96041215288228 0loss/FBt2/assignment_loss‚0.6629705939974103 1loss/SRqqt4/detection_loss‚1.5723703000389162Íó§­’ã‚(:‡ ÎØãÔþµƒ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09c’D8ä‚(:­ ÎØãÔŒûÚ * disk./.usagePercent‚15.765481574339017 network.sent‚ 5.035982e+06 disk.sda2.in‚0 disk.dm-1.out‚88.46875 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0" disk.nvme0n1.out‚ 4736.30859375 disk.dm-0.out‚ 4736.30859375 disk.drbd2080.in‚0 cpu‚25.357905467965384 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 4736.30859375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 network.recv‚744116 disk.sda1.in‚0 disk.sda.out‚88.46875 proc.cpu.threads‚17 disk.sda3.out‚88.46875 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚8.34375 disk.dm-0.in‚8.34375* proc.memory.percent‚17.216205596923828 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚8.34375 disk.rbd1.out‚ 10.474609375 disk.dm-1.in‚0( proc.memory.availableMB‚ 951.49609375$ disk./.usageGB‚234.9170684814453 disk.drbd2080.out‚0$ memory_percent‚94.19252872467041! proc.memory.rssMB‚ 2820.703125@ÏÒ8’å‚(:‡ ÝØãÔÇšŽ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙªM>;æ‚(:° ÝØãÔ“¦ë  disk.drbd2080.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 951.73828125! proc.memory.rssMB‚ 2820.703125* disk./.usagePercent‚15.765477734226005 disk.rbd0.in‚0 disk.rbd1.out‚ 10.474609375$ memory_percent‚94.19105052947998 disk.rbd0.out‚0 disk.dm-0.out‚ 4736.578125 proc.cpu.threads‚17 disk.sda.out‚ 89.01171875 disk.sda3.in‚0 disk.sda3.out‚ 89.01171875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚0 network.recv‚750351 disk.nvme0c0n1.in‚8.34375% disk./.usageGB‚234.91701126098633 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.out‚ 4736.578125 disk.drbd2265.out‚0 network.sent‚ 5.144769e+06 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.out‚ 4736.578125 disk.dm-0.in‚8.34375 disk.dm-1.out‚ 89.01171875 disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚8.34375* proc.memory.percent‚17.216205596923828 cpu‚25.35567080690904q„+>2ç- _wandbruntime‚1989  _runtime‚1989bwÇèÁ % _timestamp‚1.7884068866175208e+09 _runtime‚1989.604972044 _step‚206 & lr-AdamW/pg1‚0.0002648939020465558 & lr-AdamW/pg2‚0.0002648939020465558 trainer/global_step‚2549gÊ” ÌéÆ trainer/global_step‚2549 % _timestamp‚1.7884068866175208e+09 _runtime‚1989.604972044 _step‚206 & lr-AdamW/pg1‚0.0002648939020465558 & lr-AdamW/pg2‚0.0002648939020465558Îópg@2ê- _wandbruntime‚1989  _runtime‚1989@§D£Ö#ëÓO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2243316066304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150494466661376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164245513569280 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48167031 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144468230011904 _runtime‚1989.605945921 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2136065122304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21653604 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2136721484288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚2549 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48146994 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37109954 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148357727174144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33949723 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37109952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150494466661376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2136065122304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162002354835968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150493792296448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21653606 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146596429230229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12296551 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 _step‚207 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85256946 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2243158733312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48167031 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148357745177088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164245694842368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148357745177088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162002378776064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150493792296448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37109952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2136721484288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144468247051264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N…’ÖMFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85276985 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12296119 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146597100707341 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928öO¹sÞ'ìØO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2243158733312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48167031 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148357745177088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164245694842368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148357745177088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162002378776064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150493792296448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37109952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2136721484288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144468247051264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85276985 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12296119 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146597100707341 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2243316066304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150494466661376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164245513569280 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48167031 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144468230011904 _runtime‚1989.605945921 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2136065122304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21653604 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2136721484288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚2549 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48146994 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37109954 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148357727174144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33949723 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37109952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150494466661376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2136065122304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162002354835968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150493792296448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21653606 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146596429230229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12296551 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 _step‚207 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85256946χV¬]2í- _wandbruntime‚1990  _runtime‚1990—<ü:ã&îÝM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148414080212992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33962776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884068872935789e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150551639141376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162064005573632 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37123968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48185511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48165448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2137540925440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164308028282880 trainer/global_step‚2549 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146652821273073 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48185511 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2136884550144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162064029513728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144523151181824 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33962332 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148414080212992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2244022709248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12301002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661774 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85289416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2136884550144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85309481 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚148414098215936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12300560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚1990.281640602 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844åáZÓè&ïâM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148414080212992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33962776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884068872935789e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150551639141376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162064005573632 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37123968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48185511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48165448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2137540925440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164308028282880 trainer/global_step‚2549 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146652821273073 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48185511 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2136884550144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162064029513728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144523151181824 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33962332 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148414080212992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2244022709248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12301002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661774 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85289416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2136884550144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85309481 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚148414098215936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12300560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚1990.281640602Ðx~_»2ð- _wandbruntime‚1990  _runtime‚1990/ÚWïYñÓ .loss/FRt2/assignment_loss‚4.39003431635934 2loss/SRqqt3/assignment_loss‚1.3259022213833433 /loss/FRt1/detection_loss‚2.8593737516138287 1loss/SRqqt4/detection_loss‚2.0719201871935202 2loss/SRqqt2/assignment_loss‚1.8656134532971516 .loss/FRt4/detection_loss‚2.208864884376526 /loss/FRt3/detection_loss‚1.7718853908672667 1loss/SRqqt3/detection_loss‚1.2632085866701435 .loss/FBt1/detection_loss‚2.593135396639506 -loss/FBt4/detection_loss‚4.05691385269165 %loss/total_loss‚52.16680512915557 % _timestamp‚1.7884068872957525e+09 _runtime‚1990.283133037 /loss/FRt3/assignment_loss‚3.068819971461045 0loss/FRt4/assignment_loss‚3.7451409912109375 0loss/FBt1/assignment_loss‚0.3899624573273791 1loss/SRqqt2/detection_loss‚2.1810788379775152 trainer/global_step‚2549 1loss/SRqqt1/detection_loss‚2.6227537064716735 0loss/FRt1/assignment_loss‚4.1908534997039375 2loss/SRqqt1/assignment_loss‚2.0297245362709306 2loss/SRqqt4/assignment_loss‚1.7097464717772541 0loss/FBt2/assignment_loss‚0.5831856906407002 /loss/FBt3/detection_loss‚0.5230188287005385 epoch‚2 0loss/FBt4/assignment_loss‚1.4693515300750732 _step‚209 1loss/FBt3/assignment_loss‚0.27394103939518794 /loss/FRt2/detection_loss‚2.5010648240914217 /loss/FBt2/detection_loss‚2.4713107029596966¤uóý_òØ 2loss/SRqqt2/assignment_loss‚1.8656134532971516 .loss/FRt4/detection_loss‚2.2088ÍîêAÿ64884376526 /loss/FRt3/detection_loss‚1.7718853908672667 1loss/SRqqt3/detection_loss‚1.2632085866701435 .loss/FBt1/detection_loss‚2.593135396639506 -loss/FBt4/detection_loss‚4.05691385269165 %loss/total_loss‚52.16680512915557 % _timestamp‚1.7884068872957525e+09 _runtime‚1990.283133037 /loss/FRt3/assignment_loss‚3.068819971461045 0loss/FRt4/assignment_loss‚3.7451409912109375 0loss/FBt1/assignment_loss‚0.3899624573273791 1loss/SRqqt2/detection_loss‚2.1810788379775152 trainer/global_step‚2549 1loss/SRqqt1/detection_loss‚2.6227537064716735 0loss/FRt1/assignment_loss‚4.1908534997039375 2loss/SRqqt1/assignment_loss‚2.0297245362709306 2loss/SRqqt4/assignment_loss‚1.7097464717772541 0loss/FBt2/assignment_loss‚0.5831856906407002 /loss/FBt3/detection_loss‚0.5230188287005385 epoch‚2 0loss/FBt4/assignment_loss‚1.4693515300750732 _step‚209 1loss/FBt3/assignment_loss‚0.27394103939518794 /loss/FRt2/detection_loss‚2.5010648240914217 /loss/FBt2/detection_loss‚2.4713107029596966 .loss/FRt2/assignment_loss‚4.39003431635934 2loss/SRqqt3/assignment_loss‚1.3259022213833433 /loss/FRt1/detection_loss‚2.8593737516138287 1loss/SRqqt4/detection_loss‚2.0719201871935202ÑRâu—’ó‚(:‡ ìØãÔꙪ gpu.0.gpu‚43.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.818' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1636 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò¶”8Gô‚(:¼ ìØãÔ‡üþ  disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375" disk.dm-0.out‚4737.33056640625% disk./.usageGB‚234.91708755493164 disk.sda.in‚0 disk.sda3.out‚ 89.5234375 disk.rbd1.out‚ 10.537109375 disk.dm-0.in‚8.34375* disk./.usagePercent‚15.765482854376689 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 89.5234375 disk.drbd2265.in‚0' proc.memory.availableMB‚ 951.2890625 proc.cpu.threads‚17* proc.memory.percent‚17.216205596923828 cpu‚25.353325288338787 network.sent‚ 5.149989e+06 disk.drbd2080.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2820.703125 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2265.out‚0$ memory_percent‚94.19379234313965 disk.sda.out‚ 89.5234375 disk.drbd2266.out‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚4737.33056640625 network.recv‚754262 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚8.34375' disk.nvme0c0n1.out‚4737.33056640625 disk.dm-1.in‚0ÿ¹d’õ‚(:‡ ûØãÔÌÆ‹ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âñ˜Jö‚(:¿ ûØãÔ¨§¨  network.sent‚ 5.258376e+06* disk./.usagePercent‚15.765483366391758 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.216205596923828 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚8.34375" disk.dm-0.out‚4737.67431640625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚8.34375% disk.nvme0n1.out‚4737.67431640625 disk.sda.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 951.25390625% disk./.usageGB‚234.91709518432617 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚8.34375 disk.sda3.out‚ 90.01953125 disk.dm-1.out‚ 90.01953125 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 90.01953125$ memory_percent‚94.19400691986084 cpu‚25.35120405846504 network.recv‚760132' disk.nvme0c0n1.out‚4737.67431640625 disk.drbd1716.out‚0 disk.rbd1.out‚ 10.623046875 disk.dm-1.in‚0! proc.memory.rssMB‚ 2820.703125„ã×÷‚(:… ŠÙãÔ¨ë¦ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ný˜,Iø‚(:¾ ŠÙãÔÄ¢ü  disk.dm-1.out‚ 90.46484375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚8.34375 disk.nvme0n1.in‚8.34375! proc.memory.rssMB‚ 2820.703125 network.recv‚764252* disk./.usagePercent‚15.765487206504773 disk.drbd2265.out‚0 disk.sda2.out‚0$ memory_percent‚94.19260025024414& proc.memory.availableMB‚ 951.484375* proc.memory.percent‚17.216205596923828 cpu‚25.348887003961504 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0% disk.nvme0n1.out‚4738.12451171875 disk.dm-0.in‚8.34375 proc.cpu.threads‚17 network.sent‚ 5.264158e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚4738.12451171875 disk.rbd1.in‚0 disk.rbd1.out‚ 10.794921875 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0% disk./.usageGB‚234.91715240478516 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 90.46484375 disk.drbd2080.in‚0 disk.sda3.out‚ 90.46484375 disk.sda1.in‚0" disk.dm-0.out‚4738.12451171875 disk.dm-1.in‚0W¹R€2ù- _wandbruntime‚2026  _runtime‚2026=EÎÇúÁ & lr-AdamW/pg1‚0.0002641315497239501 & lr-AdamW/pg2‚0.0002641315497239501 trainer/global_step‚2599 % _timestamp‚1.7884069230533671e+09 _runtime‚2026.040900735 _step‚210¡5é?ÌûÆ % _timestamp‚1.7884069230533671e+09 _runtime‚2026.040900735 _step‚210 & lr-AdamW/pg1‚0.0002641315497239501 & lr-AdamW/pg2‚0.0002641315497239501 trainer/global_step‚2599ÒË ù_2ü- _wandbruntime‚2026  _runtime‚2026 ±Ø'ýÒO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153353104409600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2177050928128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167371157824000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2177707290112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚ 2026.04179954 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12522091 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153352430044672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153352430044672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12522508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2169688995485 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚419 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49073046 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚211 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153353104409600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2177050928128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2286078540288 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151175379116544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165084915659264 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167370970259456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151175397119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147213436539904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37810752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22062006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149382471097637 trainer/global_step‚2599 % _timestamp‚1.7884069230541155e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49093083 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37810754 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22062004 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2286242164736 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167371157824000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2177707290112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚ 2026.04179954 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12522091 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153352430044672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153352430044672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12522508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2169688995485 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚419 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49073046 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚211Ó¿W`2ÿ- _wandbruntime‚2026  _runtime‚2026K¡kŒ,€ÜM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147268357709824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2177870355968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2177870355968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37824768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37824768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149438863140481 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86916268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165146542456832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091500 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12526957 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86916268 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49111563 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167433483933696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86936333 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2178526731264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526539 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149438191663161 % _timestamp‚1.7884069237410042e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153410276889600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151231732155392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22070172 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167433671484928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2178526731264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49091500 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37824770 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151231750158336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153410276889600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2170505430657 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2599 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34596711 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86916268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165146542456832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091500 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12526957 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86916268 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49111563 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167433483933696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86936333 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2178526731264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526539 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149438191663161 % _timestamp‚1.7884069237410042e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153410276889600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151231732155392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22070172 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167433671484928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2178526731264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49091500 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37824770 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151231750158336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153410276889600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2170505430657 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2599 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34596711 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2177870355968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37824768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37824768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ÔV@+2‚- _wandbruntime‚2026  _runtime‚2026–TƒÎ _step‚213 0loss/FBt1/assignment_loss‚0.6015000720234139 0loss/FBt2/assignment_loss‚0.5385067239402966 /loss/FRt2/detection_loss‚2.3578706633212954 0loss/SRqqt1/detection_loss‚2.822213441133499 %loss/total_loss‚54.47296133660217 /loss/FRt2/assignment_loss‚4.067826997402102 1loss/SRqqt2/assignment_loss‚2.481072730044039 1loss/SRqqt2/detection_loss‚2.2119898621628926 0loss/FRt1/assignment_loss‚4.1092996551440315 2loss/SRqqt3/assignment_loss‚1.4547315360261843 2loss/SRqqt4/assignment_loss‚1.8037114135528867 /loss/FRt4/detection_loss‚2.5194453708827496 /loss/FBt3/detection_loss‚0.5233067428490495 /loss/FRt3/assignment_loss‚2.708086793952518 0loss/FBt3/assignment_loss‚0.2527338847182087 -loss/FRt1/detection_loss‚3.36755494429515 0loss/FRt4/assignment_loss‚3.6982020273804665 /loss/FRt3/detection_loss‚2.0879755155907738 /loss/FBt1/detection_loss‚2.5932900625116684 .loss/FBt2/detection_loss‚4.205769658088684 $ _timestamp‚1.788406923742876e+09 _runtime‚2026.730528574 2loss/SRqqt1/assignment_loss‚2.2354745906591416 0loss/FBt4/assignment_loss‚0.3096175291138934 1loss/SRqqt3/detection_loss‚1.0960301622748374 0loss/SRqqt4/detection_loss‚1.749691964764344 -loss/FBt4/detection_loss‚4.67705899477005 epoch‚2 trainer/global_step‚25996ÂvNY„Ó .loss/FBt2/detection_loss‚4.205769658088684 $ _timestamp‚1.788406923742876e+09 _runtime‚2026.730528574 2loss/SRqqt1/assignment_loss‚2.2354745906591416 0loss/FBt4/assignment_loss‚0.3096175291138934 1loss/SRqqt3/detection_loss‚1.0960301622748374 0loss/SRqqt4/detection_loss‚1.749691964764344 -loss/FBt4/detection_loss‚4.67705899477005 epoch‚2 trainer/global_step‚2599 _step‚213 0loss/FBt1/assignment_loss‚0.6015000720234139 0loss/FBt2/assignment_loss‚0.5385067239402966 /loss/FRt2/detection_loss‚2.3578706633212954 0loss/SRqqt1/detection_loss‚2.822213441133499 %loss/total_loss‚54.47296133660217 /loss/FRt2/assignment_loss‚4.067826997402102 1loss/SRqqt2/assignment_loss‚2.481072730044039 1loss/SRqqt2/detection_loss‚2.2119898621628926 0loss/FRt1/assignment_loss‚4.1092996551440315 2loss/SRqqt3/assignment_loss‚1.4547315360261843 2loss/SRqqt4/assignment_loss‚1.8037114135528867 /loss/FRt4/detection_loss‚2.5194453708827496 /loss/FBt3/detection_loss‚0.5233067428490495 /loss/FRt3/assignment_loss‚2.708086793952518 0loss/FBt3/assignment_loss‚0.2527338847182087 -loss/FRt1/detection_loss‚3.36755494429515 0loss/FRt4/assignment_loss‚3.6982020273804665 /loss/FRt3/detection_loss‚2.0879755155907738 /loss/FBt1/detection_loss‚2.5932900625116684ÕÅ:J›…‚(:… ™ÙãÔä“­ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.26' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù÷›H†‚(:½ ™ÙãÔº¶ÿ  network.recv‚768482* disk./.usagePercent‚15.765483366391758& disk.nvme0c0n1.out‚4739.0263671875 disk.dm-0.in‚8.34375 disk.sda3.out‚ 91.05078125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 91.05078125 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 951.22265625 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd1.out‚ 10.794921875 disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 cpu‚25.346594238920023 disk.nvme0n1.in‚8.34375$ disk.nvme0n1.out‚4739.0263671875 disk.sda2.in‚0% disk./.usageGB‚234.91709518432617 disk.sda.out‚ 91.05078125! disk.dm-0.out‚4739.0263671875$ memory_percent‚94.19419765472412 network.sent‚ 5.270416e+06 disk.nvme0c0n1.in‚8.34375 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2820.703125* proc.memory.percent‚17.216205596923828 disk.drbd1716.in‚0 disk.sda1.out‚0lõŒù’‡‚(:‡ ¨ÙãÔá¡ö gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÀIˆ‚(:¾ ¨ÙãÔï­ø ' proc.memory.availableMB‚ 477.4453125 network.sent‚ 5.378119e+06* disk./.usagePercent‚15.765561448689699 disk.dm-0.in‚ 8.3515625 disk.sda.in‚0 disk.sda.out‚ 93.2578125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 93.2578125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2820.703125* proc.memory.percent‚17.216205596923828 network.recv‚774175$ disk.nvme0n1.out‚4748.8134765625 disk.sda3.in‚0 disk.rbd1.out‚ 10.857421875$ memory_percent‚97.08590507507324 disk.nvme0n1.in‚ 8.3515625 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 cpu‚25.344551654330633$ disk./.usageGB‚234.9182586669922! disk.dm-0.out‚4748.8134765625 disk.dm-1.in‚0 disk.dm-1.out‚ 93.2578125 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 8.3515625& disk.nvme0c0n1.out‚4748.8134765625…¢+ÿ2‰- _wandbruntime‚2062  _runtime‚2062:W„TÇŠÁ & lr-AdamW/pg1‚0.0002633428137291284 & lr-AdamW/pg2‚0.0002633428137291284 trainer/global_step‚2649 % _timestamp‚1.7884069593136694e+09 _runtime‚2062.301101774 _step‚214SÁ¿Ì‹Æ & lr-AdamW/pg2‚0.0002633428137291284 trainer/global_step‚2649 % _timestamp‚1.7884069593136694e+09 _runtime‚2062.301101774 _step‚214 & lr-AdamW/pg1‚0.0002633428137291284Ö;€ 2Œ- _wandbruntime‚2062  _runtime‚2062v—¥ŠØ'ÒO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35220614 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50019135 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38511554 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149958660107264 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788406959314391e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2329075896832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149958643067904 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156211067792896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152169184442157 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218693095936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153993049061888 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚2062.302044544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218036733952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168167428602368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156211742157824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218036733952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156211067792896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2328920660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2210524334893 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153993049061888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153993031058944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218693095936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22470406 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50019135 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49999098 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153993031058944òš~™Ž×O PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38511552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35220176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2209869897141 trainer/global_step‚2649 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22470404 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152168512965045 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88530689 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚2062.302044544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZH4|îDDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218036733952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168167428602368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156211742157824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218036733952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156211067792896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2328920660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2210524334893 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153993049061888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153993031058944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218693095936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22470406 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50019135 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49999098 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153993031058944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35220614 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50019135 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38511554 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149958660107264 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788406959314391e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2329075896832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149958643067904 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156211067792896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152169184442157 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218693095936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153993049061888 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88563185 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170558850052608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38525568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154049384097792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154049384097792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88543120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12754237 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168229103280128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154049402100736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152224905007889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2329770712576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156268914637824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50017552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168229079340032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154049402100736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2218856161792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884069599997563e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150013564237824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2211340770065 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38525570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22478572 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35233244 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156268914637824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2218856161792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2210686332105 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219512537088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38525568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38525570 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219512537088 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35232809 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2329940615168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50037615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170559043895296 trainer/global_step‚2649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12754670 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚2062.987638999 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚2062.987638999 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88563185 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170558850052608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38525568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154049384097792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154049384097792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88543120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12754237 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168229103280128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154049402100736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152224905007889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2329770712576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156268914637824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50017552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168229079340032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154049402100736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2218856161792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884069599997563e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150013564237824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2211340770065 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38525570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22478572 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35233244 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156268914637824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2218856161792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2210686332105 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219512537088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38525568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38525570 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219512537088 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35232809 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430Ø’dêŒ2’- _wandbruntime‚2062  _runtime‚2062ºÈmŒV“Ð epoch‚2 /loss/FRt3/assignment_loss‚2.509387140688689 2loss/SRqqt1/assignment_loss‚2.0813006346042338 0loss/SRqqt1/detection_loss‚2.768466967802781 -loss/FRt4/detection_loss‚2.47015604906612 /loss/FBt3/detection_loss‚0.7393905041741199 _step‚217 0loss/FRt1/assignment_loss‚3.5664513234434456 /loss/FRt2/assignment_loss‚3.997606036337939 /loss/FRt4/assignment_loss‚4.266195820437537 2loss/SRqqt2/assignment_loss‚1.7022614711523056 .loss/FRt2/detection_loss‚2.316390886225484 0loss/FBt1/assignment_loss‚0.6437764714161555 0loss/FBt2/assignment_loss‚0.5622268691658567 .loss/FBt4/detection_loss‚4.659652471542358 .loss/FBt1/detection_loss‚3.694551706314087 trainer/global_step‚2649 2loss/SRqqt3/assignment_loss‚1.7393341319341409 /loss/FRt3/detection_loss‚2.0984061716691307 $ _timestamp‚1.788406960001342e+09 _runtime‚2062.988720642 2loss/SRqqt4/assignment_loss‚2.0406596194952726 0loss/FBt3/assignment_loss‚0.2997756862408412 1loss/FBt4/assignment_loss‚0.20141470432281494 .loss/FRt1/detection_loss‚2.974344985238437 1loss/SRqqt3/detection_loss‚1.4305277389653943 1loss/SRqqt4/detection_loss‚1.7687751536185925 /loss/FBt2/detection_loss‚2.9022875002452304 1loss/SRqqt2/detection_loss‚1.9681197661161423 &loss/total_loss‚53.401459810217105;­J‚[”Õ .loss/FBt1/detection_loss‚3.694551706314087 trainer/global_step‚2649 2loss/SRqqt3/assignment_loss‚1.7393341319341409 /loss/FRt3/detection_loss‚2.0984061716691307 $ _timestamp‚1.788406960001342e+09 _runtime‚2062.988720642 2loss/SRqqt4/assignment_loss‚2.0406596194952726 0loss/FBt3/assignment_loss‚0.2997756862408412 1loss/FBt4/assignment_loss‚0.20141470432281494 .loss/FRt1/detection_loss‚2.974344985238437 1loss/SRqqt3/detection_loss‚1.4305277389653943 1loss/SRqqt4/detection_loss‚1.7687751536185925 /loss/FBt2/detection_loss‚2.9022875002452304 1loss/SRqqt2/detection_loss‚1.9681197661161423 &loss/total_loss‚53.401459810217105 epoch‚2 /loss/FRt3/assignment_loss‚2.509387140688689 2loss/SRqqt1/assignment_loss‚2.0813006346042338 0loss/SRqqt1/detection_loss‚2.768466967802781 -loss/FRt4/detection_loss‚2.47015604906612 /loss/FBt3/detection_loss‚0.7393905041741199 _step‚217 0loss/FRt1/assignment_loss‚3.5664513234434456 /loss/FRt2/assignment_loss‚3.997606036337939 /loss/FRt4/assignment_loss‚4.266195820437537 2loss/SRqqt2/assignment_loss‚1.7022614711523056 .loss/FRt2/detection_loss‚2.316390886225484 0loss/FBt1/assignment_loss‚0.6437764714161555 0loss/FBt2/assignment_loss‚0.5622268691658567 .loss/FBt4/detection_loss‚4.659652471542358Ù¤l$-’•‚(:‡ ·ÙãÔ•à³ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.536' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f|ÎæM–‚(: ·ÙãÔå‰þ  disk.rbd1.out‚ 10.857421875 disk.dm-1.out‚ 93.96484375 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 8.3515625 disk.drbd1716.out‚0$ disk.nvme0n1.out‚4749.0869140625! disk.dm-0.out‚4749.0869140625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 8.3515625 disk.dm-1.in‚0 disk.sda.out‚ 93.96484375 disk.drbd2266.out‚0* proc.memory.percent‚17.216205596923828 proc.cpu.threads‚17* disk./.usagePercent‚15.765627242625998% disk./.usageGB‚234.91923904418945 disk.dm-0.in‚ 8.3515625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚97.08876609802246! proc.memory.rssMB‚ 2820.703125 cpu‚25.342347349968726 network.sent‚ 5.384092e+06 network.recv‚778371 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0& disk.nvme0c0n1.out‚4749.0869140625 disk.sda3.out‚ 93.96484375 disk.drbd2080.out‚0' proc.memory.availableMB‚ 476.9765625Nm‰—‚(:’ ÆÙãÔ¸°˜ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.173' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.034600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@t~˜‚(:¿ ÆÙãÔµÿ7¯ú¯  disk.drbd1716.in‚0 disk.sda.out‚ 94.4765625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0$ memory_percent‚97.09033966064453 disk.sda2.out‚0% disk.nvme0n1.out‚4752.29248046875 disk.dm-1.out‚ 94.4765625 disk.drbd2265.in‚0 disk.sda3.out‚ 94.4765625 disk.drbd2266.in‚0% proc.memory.availableMB‚ 476.71875 disk.sda1.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚4752.29248046875 disk.nvme0n1.in‚ 8.3515625 cpu‚25.34023451292712 network.sent‚ 5.492389e+06 network.recv‚784513% disk./.usageGB‚234.91921997070312 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2820.703125 disk.dm-0.in‚ 8.3515625" disk.dm-0.out‚4752.29248046875 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 11.115234375 disk.dm-1.in‚0 disk.sda.in‚0* proc.memory.percent‚17.216205596923828* disk./.usagePercent‚15.765625962588329 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 8.35156259¤Ù2™- _wandbruntime‚2098  _runtime‚2098+AjÛÆšÀ & lr-AdamW/pg1‚0.0002625278584334899 & lr-AdamW/pg2‚0.0002625278584334899 trainer/global_step‚2699 $ _timestamp‚1.788406995409796e+09 _runtime‚2098.397240118 _step‚218䩋˛Š_step‚218 & lr-AdamW/pg1‚0.0002625278584334899 & lr-AdamW/pg2‚0.0002625278584334899 trainer/global_step‚2699 $ _timestamp‚1.788406995409796e+09 _runtime‚2098.397240118Ú¥=rÄ2œ- _wandbruntime‚2098  _runtime‚2098ÿ¦}Ù'ÓO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90137502 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2259678901760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90157541 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50925150 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173621759115264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159070379906048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2259022539776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2259022539776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35854439 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156810683001344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156810701004288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156810683001344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50945187 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚2098.398162058 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171249965485568 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159069705541120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39212354 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39212352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2371793629696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251359674301 _step‚219 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35854863 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173621944582656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159069705541120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154955226309565 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50925150 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159070379906048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152703866635264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467392 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159069705541120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154955226309565 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50925150 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159070379906048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152703866635264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50945187 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚2098.398162058 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171249965485568 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159069705541120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39212354 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39212352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2371793629696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251359674301 _step‚219 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35854863 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173621944582656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35867094 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22886974 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39226370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259841967616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171311616223232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50943604 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2372811588096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2260498342912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156867036040192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260498342912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39226368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2251521671513 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39226368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159127552386048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50943604 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2372641685504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78840699608101e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156867036040192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156867054043136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173684257908736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39226370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12980566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 _step‚220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50963667 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159126878007808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2699 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50963667 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22886972 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35867540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚2099.069161818 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259841967616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021¬»ÞÄ¡àM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35867540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚2099.069161818 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259841967616 FDeviceStatsMonitor.on_train_batcÒqµ"$h_end/requested_bytes.all.peak‚ 9643699021 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35867094 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22886974 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39226370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259841967616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171311616223232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50943604 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2372811588096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2260498342912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156867036040192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260498342912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39226368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2251521671513 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39226368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159127552386048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50943604 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2372641685504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78840699608101e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156867036040192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156867054043136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173684257908736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39226370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12980566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 _step‚220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50963667 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159126878007808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2699 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50963667 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22886972 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844ÜE×v2¢- _wandbruntime‚2099  _runtime‚2099Ù’¹òV£Ð /loss/FRt1/assignment_loss‚4.342034581634733 0loss/FBt2/assignment_loss‚0.5289654121734197 1loss/SRqqt4/detection_loss‚1.8834075156984658 1loss/FBt1/assignment_loss‚0.49210691038751975 /loss/FRt3/detection_loss‚1.9037264913320542 0loss/SRqqt1/detection_loss‚2.393751815387181 1loss/SRqqt2/detection_loss‚2.0570956285704267 .loss/FBt2/detection_loss‚1.980912771075964 /loss/FBt3/detection_loss‚0.5411747949555482 _runtime‚2099.070295688 /loss/FRt2/assignment_loss‚4.559107550259294 0loss/FBt3/assignment_loss‚0.2937507373161589 /loss/FRt1/detection_loss‚2.7568212615119085 .loss/FRt2/detection_loss‚3.338280793921701 %loss/total_loss‚50.67623456319994 trainer/global_step‚2699 /loss/FBt4/detection_loss‚2.9392115473747253 .loss/FRt4/assignment_loss‚4.12138504196297 2loss/SRqqt1/assignment_loss‚2.2635689394814626 2loss/SRqqt4/assignment_loss‚1.3946119599558155 /loss/FBt1/detection_loss‚2.8206230774521828 epoch‚2 _step‚221 0loss/FRt3/assignment_loss‚1.8485769358548252 1loss/SRqqt2/assignment_loss‚2.035299984907562 0loss/FBt4/assignment_loss‚1.1493776738643646 0loss/SRqqt3/detection_loss‚1.042846193819335 2loss/SRqqt3/assignment_loss‚1.4813090674579144 .loss/FRt4/detection_loss‚2.508287876844406 % _timestamp‚1.7884069960829434e+09iýñ&[¤Õ _runtime‚2099.070295688 /loss/FRt2/assignment_loss‚4.559107550259294 0loss/FBt3/assignment_loss‚0.2937507373161589 /loss/FRt1/detection_loss‚2.7568212615119085 .loss/FRt2/detection_loss‚3.338280793921701 %loss/total_loss‚50.67623456319994 trainer/global_step‚2699 /loss/FBt4/detection_loss‚2.9392115473747253 .loss/FRt4/assignment_loss‚4.12138504196297 2loss/SRqqt1/assignment_loss‚2.2635689394814626 2loss/SRqqt4/assignment_loss‚1.3946119599558155 /loss/FBt1/detection_loss‚2.8206230774521828 epoch‚2 _step‚221 0loss/FRt3/assignment_loss‚1.8485769358548252 1loss/SRqqt2/assignment_loss‚2.035299984907562 0loss/FBt4/assignment_loss‚1.1493776738643646 0loss/SRqqt3/detection_loss‚1.042846193819335 2loss/SRqqt3/assignment_loss‚1.4813090674579144 .loss/FRt4/detection_loss‚2.508287876844406 % _timestamp‚1.7884069960829434e+09 /loss/FRt1/assignment_loss‚4.342034581634733 0loss/FBt2/assignment_loss‚0.5289654121734197 1loss/SRqqt4/detection_loss‚1.8834075156984658 1loss/FBt1/assignment_loss‚0.49210691038751975 /loss/FRt3/detection_loss‚1.9037264913320542 0loss/SRqqt1/detection_loss‚2.393751815387181 1loss/SRqqt2/detection_loss‚2.0570956285704267 .loss/FBt2/detection_loss‚1.980912771075964 /loss/FBt3/detection_loss‚0.5411747949555482ÝÄÙ±’¥‚(:‡ ÕÙãÔ姜 gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.066' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?¬sO¦‚(:Ä ÕÙãÔÔ…Ž  disk.nvme0c0n1.in‚ 8.3515625 disk.rbd1.out‚ 11.115234375 proc.cpu.threads‚17 network.sent‚ 5.498648e+06* disk./.usagePercent‚15.765625962588329 disk.dm-1.out‚ 95.02734375 disk.sda2.in‚0 disk.nvme0n1.in‚ 8.3515625% disk.nvme0n1.out‚4752.56201171875" disk.dm-0.out‚4752.56201171875% disk./.usageGB‚234.91921997070312 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.sda.out‚ 95.02734375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚4752.56201171875 disk.dm-0.in‚ 8.3515625' proc.memory.availableMB‚ 476.3984375 disk.dm-1.in‚0 disk.sda3.out‚ 95.02734375 disk.rbd1.in‚0* proc.memory.percent‚17.216205596923828 network.recv‚789019 disk.drbd2265.out‚0 disk.drbd1716.out‚0$ memory_percent‚97.09229469299316 disk.drbd2265.in‚0! proc.memory.rssMB‚ 2820.703125 disk.sda3.in‚0 disk.drbd2080.out‚0 cpu‚25.33824205478745 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0£±¦§‚(:„ äÙãÔ®¯“ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚146.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.39 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m§}0?¨‚(:´ äÙãÔ†ö‰  network.sent‚ 5.606264e+06' disk./.usagePercent‚15.832675359836* proc.memory.percent‚17.216205596923828 disk.sda3.in‚0 disk.sda3.out‚ 95.5703125 disk.sda2.out‚0! disk.nvme0n1.out‚ 4994.4453125 disk.dm-1.in‚0 disk.rbd1.out‚ 11.177734375 disk.dm-0.out‚ 4994.4453125 disk.dm-1.out‚ 95.5703125 disk.sda1.out‚0 network.recv‚794640 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 8.37890625 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 476.8828125 cpu‚25.33608088097933 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 8.37890625# disk.nvme0c0n1.out‚ 4994.4453125 disk.nvme0n1.in‚ 8.37890625% disk./.usageGB‚235.91830444335938 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.sda.out‚ 95.5703125# memory_percent‚97.0893383026123 disk.drbd2265.in‚0! proc.memory.rssMB‚ 2820.703125 disk.rbd0.in‚0ê*Ã]©‚(:’ óÙãÔü…• gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.355' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.270999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹:úÏOª‚(:Ä óÙãÔÒåæ  cpu‚25.33409638901118 disk.rbd0.in‚0& disk.nvme0c0n1.out‚6112.5029296875 disk.rbd1.out‚ 11.177734375# memory_percent‚97.0893144607544 network.recv‚798777 disk.drbd2080.out‚0 disk.dm-1.out‚ 96.1640625 disk.sda3.in‚0( proc.memory.availableMB‚ 476.88671875 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0* proc.memory.percent‚17.216205596923828% disk./.usageGB‚236.24456024169922 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0! proc.memory.rssMB‚ 2820.703125 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 96.1640625* disk./.usagePercent‚15.854570660209433 disk.rbd0.out‚0 disk.sda2.out‚0! disk.dm-0.out‚6112.5029296875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 96.1640625 proc.cpu.threads‚17 network.sent‚ 5.612328e+06 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 11.45703125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 11.45703125$ disk.nvme0n1.out‚6112.5029296875 disk.dm-0.in‚ 11.45703125.[2«- _wandbruntime‚2134  _runtime‚2134)ŒÆ¬À & lr-AdamW/pg2‚0.0002616868536724968 trainer/global_step‚2749 $ _timestamp‚1.788407031511836e+09 _runtime‚2134.499267435 _step‚222 & lr-AdamW/pg1‚0.0002616868536724968&®ýË­Å _runtime‚2134.499267435 _step‚222 & lr-AdamW/pg1‚0.0002616868536724968 & lr-AdamW/pg2‚0.0002616868536724968 trainer/global_step‚2749 $ _timestamp‚1.788407031511836e+09Þ²˜±„2®- _wandbruntime‚2134  _runtime‚2134L‚Ù'¯ÓO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36489860 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36489435 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39913152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161929017654272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2300008345600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159628334943744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161929017654272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13202231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176747369007616 % _timestamp‚1.7884070315125396e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39913152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51871239 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2300008345600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚2749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39913154 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39913154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51851202 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2414842698752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159628352946688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91784393 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23287206 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13202654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2291540575957 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155449056123904 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51871239 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161928343289344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157741268176973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161928343289344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155449073163264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚223 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2300664707584 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2300664707584 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159628352946688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157740596699861í0Ò"°ØO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155449056123904 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51871239 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161928343289344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157741268176973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161928343289344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155449073163264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚223 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2300664707584 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2300664707584 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.alš{_¼located‚159628352946688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157740596699861 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36489860 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36489435 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39913152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161929017654272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2300008345600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159628334943744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161929017654272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13202231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176747369007616 % _timestamp‚1.7884070315125396e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39913152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51871239 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2300008345600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚2749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39913154 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39913154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51851202 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2414842698752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159628352946688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91784393 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23287206 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13202654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2291540575957 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ßúããµ2±- _wandbruntime‚2135  _runtime‚2135ØÐQ²ã&²ÝM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2300827773440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176809677238272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157796317265385 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155503977293824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2301484148736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13207203 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159684687982592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36502577 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161985515756032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161986190134272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176809866886656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161986190134272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51889719 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39927168 % _timestamp‚1.7884070322074862e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39927168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174394177046528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2415524131840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91796824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51889719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23295374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2415689840128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39927170 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚2749 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2300827773440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36502136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13206764 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174394153106432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91816889 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2292357010921 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚399271704ðbhè&³âM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2415524131840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91796824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51889719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23295374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2415689840128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39927170 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚2749 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2300827773440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36502136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13206764 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174394153106432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91816889 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2292357010921 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39927170 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2300827773440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176809677238272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157796317265385 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155503977293824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2301484148736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13207203 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159684687982592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36502577 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161985515756032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161986190134272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176809866886656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161986190134272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51889719 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39927168 % _timestamp‚1.7884070322074862e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39927168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174394177046528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360àfzHj2´- _wandbruntime‚2135  _runtime‚2135èÔóŽWµÑ -loss/FRt2/detection_loss‚2.56543742321633 /loss/FRt4/detection_loss‚2.7767669210831323 0loss/SRqqt4/detection_loss‚1.706833508171019 .loss/FBt2/detection_loss‚2.629780512589675 trainer/global_step‚2749 2loss/SRqqt3/assignment_loss‚1.2178449070797517 1loss/FBt3/assignment_loss‚0.24739682309177113 1loss/SRqqt2/detection_loss‚1.9538653489540925 /loss/FBt4/detection_loss‚2.8551892723355974 0loss/FRt2/assignment_loss‚3.3504086330130294 /loss/FRt3/assignment_loss‚1.618722659090291 2loss/SRqqt4/assignment_loss‚1.9506941743072916 /loss/FBt2/assignment_loss‚0.597515894816402 &loss/total_loss‚49.219771218918126 2loss/SRqqt2/assignment_loss‚1.7633319708772681 /loss/FBt1/detection_loss‚3.5701367259025574 _step‚225 2loss/SRqqt1/assignment_loss‚2.7896777987480164 0loss/SRqqt1/detection_loss‚2.419899359345436 _runtime‚ 2135.19691481 .loss/FRt1/assignment_loss‚4.34479770436883 0loss/FBt4/assignment_loss‚0.3424314727599267 epoch‚2 0loss/FRt4/assignment_loss‚3.0909064180321164 /loss/FRt1/detection_loss‚3.0914591401815414 /loss/FRt3/detection_loss‚1.7711606382027916 1loss/SRqqt3/detection_loss‚1.1960841338221844 /loss/FBt3/detection_loss‚0.8199612285119171 % _timestamp‚1.7884070322095523e+09 0loss/FBt1/assignment_loss‚0.5494685504171584¡ Ò\¶Ö 0loss/FRt2/assignment_loss‚3.3504086330130294 /loss/FRt3/assignment_loss‚1.618722659090291 2loss/SRqqt4/assignment_loss‚1.9506941743072916 /loss/FBt2/assignment_loss‚0.597515894816402 &loss/total_loss‚49.219771218918126 2loss/SRqqt2/assignment_loss‚1.7633319708772681 /loss/FBt1/detection_loss‚3.5701367259025574 _step‚225 2loss/SRqqt1/assignment_loss‚2.7896777987480164 0loss/SRqqt1/detection_loss‚2.419899359345436 _runtime‚ 2135.19691481 .loss/FRt1/assignment_loss‚4.34479770436883 0loss/FBt4/assignment_loss‚0.3424314727599267 epoch‚2 0loss/FRt4/assignment_loss‚3.0909064180321164 /loss/FRt1/detection_loss‚3.0914591401815414 /loss/FRt3/detection_loss‚1.7711606382027916 1loss/SRqqt3/detection_loss‚1.1960841338221844 /loss/FBt3/detection_loss‚0.8199612285119171 % _timestamp‚1.7884070322095523e+09 0loss/FBt1/assignment_loss‚0.5494685504171584 -loss/FRt2/detection_loss‚2.56543742321633 /loss/FRt4/detection_loss‚2.7767669210831323 0loss/SRqqt4/detection_loss‚1.706833508171019 .loss/FBt2/detection_loss‚2.629780512589675 trainer/global_step‚2749 2loss/SRqqt3/assignment_loss‚1.2178449070797517 1loss/FBt3/assignment_loss‚0.24739682309177113 1loss/SRqqt2/detection_loss‚1.9538653489540925 /loss/FBt4/detection_loss‚2.8551892723355974áðì“¡’·‚(:‡ ‚ÚãÔТ› gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.387' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8774 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02 ä0N¸‚(:à ‚ÚãÔœ‚¶ $ disk.nvme0n1.out‚6115.8818359375! disk.dm-0.out‚6115.8818359375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 12.51953125) disk./.usagePercent‚15.85454505945601 disk.sda.in‚0 disk.sda.out‚ 98.734375 disk.sda2.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.216205596923828 network.sent‚ 5.618231e+06 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚236.24417877197266 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 477.09765625! proc.memory.rssMB‚ 2820.703125 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 12.51953125 disk.sda3.out‚ 98.734375! disk.nvme0c0n1.in‚ 12.51953125 disk.drbd1716.out‚0$ memory_percent‚97.08802700042725 cpu‚25.332208518448475 disk.dm-1.out‚ 98.734375 disk.drbd2080.out‚0& disk.nvme0c0n1.out‚6115.8818359375 disk.rbd1.out‚ 11.4052734375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.recv‚803297 disk.drbd2265.out‚0 disk.rbd0.out‚0vÎË’¹‚(:‡ ‘ÚãÔÌߤ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘›häQº‚(:Æ ‘ÚãÔ³êì  cpu‚25.33019461072108% disk./.usageGB‚236.24412155151367 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2820.703125 network.sent‚ 5.726985e+06 network.recv‚809426* disk./.usagePercent‚15.854541219342996 disk.sda3.out‚ 100.53125" disk.dm-0.out‚6120.65576171875$ memory_percent‚97.09093570709229( proc.memory.availableMB‚ 476.62109375* proc.memory.percent‚17.216205596923828 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 12.51953125 disk.sda.out‚ 100.53125 disk.drbd2266.in‚0 disk.drbd2080.out‚0% disk.nvme0n1.out‚6120.65576171875 disk.sda.in‚0' disk.nvme0c0n1.out‚6120.65576171875 disk.drbd1716.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 11.4990234375 disk.dm-0.in‚ 12.51953125 disk.dm-1.in‚0 disk.dm-1.out‚ 100.53125 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 12.51953125 disk.rbd1.in‚0¬3°ü2»- _wandbruntime‚2170  _runtime‚2170ÔOB¡Ç¼Á % _timestamp‚1.7884070677370043e+09 _runtime‚2170.724459806 _step‚226 & lr-AdamW/pg1‚0.0002608199747102808 & lr-AdamW/pg2‚0.0002608199747102808 trainer/global_step‚2799'æÆÌ½Æ & lr-AdamW/pg1‚0.0002608199747102808 & lr-AdamW/pg2‚0.0002608199747102808 trainer/global_step‚2799 % _timestamp‚1.7884070677370043e+09 _runtime‚2170.724459806 _step‚226â0ª#2¾- _wandbruntime‚2170  _runtime‚21706vh58¿ÓO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2340994151424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162446004889088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2341650513408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 BDeviceStatsMonitor.on_train_batch_s÷©@d¡!tart/active.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93411245 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164786981037568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162445986886144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164787655402496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164787655402496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13429693 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2457755935232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2333030353117 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚227 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177415039251968 % _timestamp‚1.7884070677377603e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37125297 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179872819127296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158194279691264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162445986886144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160527310044381 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93391206 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52797291 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2457586019328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52777254 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168»bÉÞ'ÀØO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162446004889088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2341650513408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93411245 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164786981037568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162445986886144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164787655402496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164787655402496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13429693 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2457755935232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2333030353117 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚227 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177415039251968 % _timestamp‚1.7884070677377603e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37125297 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179872819127296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158194279691264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162445986886144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160527310044381 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93391206 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52797291 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2457586019328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52777254 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2340994151424ãô+÷2Á- _wandbruntime‚2171  _runtime‚2171ý¹û7ã&ÂÝM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2341813579264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884070684139156e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52815771 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164844827882496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37138360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2333192350329 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2333846788289 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162502357927936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40627970 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52815771 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37137907 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23703772 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93423676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177476713929728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40627968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341813579264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52795708 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342469954560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160583030610113 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164844827882496 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40627968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179935134786048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160582359132793 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158249183821824åÛA´IÃâM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2333846788289 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162502357927936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40627970 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52815771 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37137907 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23703772 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93423676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177476713929728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40627968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341813579264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52795708 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342469954560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160583030610113 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164844827882496 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40627968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179935134786048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160582359132793 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158249183821824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2341813579264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884070684139156e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52815771 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164844827882496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37138360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2333192350329 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2äh€(2Ä- _wandbruntime‚2171  _runtime‚2171¾àýmVÅÐ _runtime‚2171.403248181 0loss/FBt4/assignment_loss‚0.6750078002611796 _step‚229 /loss/FRt1/assignment_loss‚4.036658028761546 0loss/SRqqt1/detection_loss‚2.459314674139023 /loss/FBt1/detection_loss‚3.2525283992290497 .loss/FBt2/detection_loss‚2.778139960765839 %loss/total_loss‚49.36109300828937 epoch‚2 0loss/FRt3/assignment_loss‚1.8581716845620353 2loss/SRqqt4/assignment_loss‚1.9479754247209604 0loss/FBt1/assignment_loss‚0.4410599794768351 /loss/FRt4/detection_loss‚2.7254356969343987 1loss/SRqqt2/detection_loss‚2.0720416002652864 /loss/FBt3/detection_loss‚0.7589338366784795 /loss/FRt2/assignment_loss‚3.666224161783854 2loss/SRqqt2/assignment_loss‚1.6167004965245724 1loss/SRqqt3/detection_loss‚1.2502304586581885 1loss/SRqqt4/detection_loss‚1.8161334952010828 # _timestamp‚1.78840706841583e+09 0loss/FRt4/assignment_loss‚3.9488290611066317 /loss/FRt1/detection_loss‚2.9060416917006173 /loss/FRt2/detection_loss‚2.4487319456206427 .loss/FRt3/detection_loss‚1.911159560927805 /loss/FBt4/detection_loss‚2.0921143045028052 1loss/SRqqt1/assignment_loss‚2.527733447797158 1loss/SRqqt3/assignment_loss‚1.293536220630631 0loss/FBt2/assignment_loss‚0.6047844991087914 0loss/FBt3/assignment_loss‚0.2736065789319607 trainer/global_step‚2799?Ãä[ÆÕ 1loss/SRqqt1/assignment_loss‚2.527733447797158 1loss/SRqqt3/assignment_loss‚1.293536220630631 0loss/FBt2/assignment_loss‚0.6047844991087914 0loss/FBt3/assignment_loss‚0.2736065789319607 trainer/global_step‚2799 _runtime‚2171.403248181 0loss/FBt4/assignment_loss‚0.6750078002611796 _step‚229 /loss/FRt1/assignment_loss‚4.036658028761546 0loss/SRqqt1/detection_loss‚2.459314674139023 /loss/FBt1/detection_loss‚3.2525283992290497 .loss/FBt2/detection_loss‚2.778139960765839 %loss/total_loss‚49.36109300828937 epoch‚2 0loss/FRt3/assignment_loss‚1.8581716845620353 2loss/SRqqt4/assignment_loss‚1.9479754247209604 0loss/FBt1/assignment_loss‚0.4410599794768351 /loss/FRt4/detection_loss‚2.7254356969343987 1loss/SRqqt2/detection_loss‚2.0720416002652864 /loss/FBt3/detection_loss‚0.7589338366784795 /loss/FRt2/assignment_loss‚3.666224161783854 2loss/SRqqt2/assignment_loss‚1.6167004965245724 1loss/SRqqt3/detection_loss‚1.2502304586581885 1loss/SRqqt4/detection_loss‚1.8161334952010828 # _timestamp‚1.78840706841583e+09 0loss/FRt4/assignment_loss‚3.9488290611066317 /loss/FRt1/detection_loss‚2.9060416917006173 /loss/FRt2/detection_loss‚2.4487319456206427 .loss/FRt3/detection_loss‚1.911159560927805 /loss/FBt4/detection_loss‚2.0921143045028052åóHì’Ç‚(:‡  ÚãÔÇÖˆ gpu.0.gpu‚43.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ä+©UÈ‚(:Ê  ÚãÔ±•“  disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 101.06640625$ memory_percent‚97.09110260009766 disk.dm-1.out‚ 101.06640625 disk.rbd0.out‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚6123.8974609375! proc.memory.rssMB‚ 2820.703125* disk./.usagePercent‚15.854541219342996% disk./.usageGB‚236.24412155151367 disk.drbd2266.in‚0 disk.drbd2080.in‚0 network.sent‚ 5.732222e+06 network.recv‚812968 disk.sda3.out‚ 101.06640625 disk.sda1.out‚0& disk.nvme0c0n1.out‚6123.8974609375 disk.nvme0n1.in‚ 12.51953125 disk.dm-0.in‚ 12.51953125% proc.memory.availableMB‚ 476.59375 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0* proc.memory.percent‚17.216205596923828 disk.sda3.in‚0 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 11.5615234375 disk.sda.in‚0! disk.nvme0c0n1.in‚ 12.51953125 disk.drbd1716.in‚0 cpu‚25.328163339136022 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0! disk.dm-0.out‚6123.8974609375¯²2É‚(:’ ¯ÚãԋƧ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.328' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.665600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07HëXÊ‚(:Í ¯ÚãÔ•Ïì  network.recv‚819173 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 101.65234375 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚6124.51611328125 disk.dm-0.in‚ 12.51953125 disk.dm-1.out‚ 101.65234375 disk.sda.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚97.09148406982422! proc.memory.rssMB‚ 2820.703125 proc.cpu.threads‚17 disk.rbd0.in‚0* disk./.usagePercent‚15.854549667591625! disk.nvme0c0n1.in‚ 12.51953125 disk.rbd1.in‚0" disk.dm-0.out‚6124.51611328125 disk.sda3.in‚0 disk.drbd2266.out‚0% disk./.usageGB‚236.24424743652344 disk.dm-1.in‚0% proc.memory.availableMB‚ 476.53125 disk.drbd1716.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 cpu‚25.326399715532325 network.sent‚ 5.840943e+06 disk.drbd1716.out‚0 disk.rbd1.out‚ 11.5615234375 disk.nvme0n1.in‚ 12.51953125% disk.nvme0n1.out‚6124.51611328125 disk.sda3.out‚ 101.65234375 disk.drbd2080.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.216205596923828+—.’Ë‚(:‡ ¾ÚãÔŽœœ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h0@ÉUÌ‚(:Ê ¾ÚãÔÕé€ ! proc.memory.rssMB‚ 2820.703125* proc.memory.percent‚17.216205596923828 cpu‚25.324448514611248 disk.sda.in‚0 disk.sda.out‚ 104.3671875 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚15.854635942130665 disk.dm-0.in‚ 12.52734375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 12.52734375 disk.dm-1.in‚0 disk.dm-1.out‚ 104.3671875 disk.sda3.out‚ 104.3671875 disk.sda2.out‚0 disk.drbd1716.out‚0$ memory_percent‚97.08998203277588% disk./.usageGB‚236.24553298950195 disk.drbd1716.in‚0% disk.nvme0n1.out‚6134.69580078125 disk.drbd2266.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 12.52734375 network.recv‚823655" disk.dm-0.out‚6134.69580078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 476.77734375 network.sent‚ 5.846777e+06' disk.nvme0c0n1.out‚6134.69580078125 disk.rbd1.out‚ 11.8515625Óÿ‡d2Í- _wandbruntime‚2206  _runtime‚2206½ç¥ÇÎÁ & lr-AdamW/pg1‚0.0002599274022031181 & lr-AdamW/pg2‚0.0002599274022031181 trainer/global_step‚2849 % _timestamp‚1.7884071036436167e+09 _runtime‚2206.631086263 _step‚230‘VàJÌÏÆ & lr-AdamW/pg2‚0.0002599274022031181 trainer/global_step‚2849 % _timestamp‚1.7884071036436167e+09 _runtime‚2206.631086263 _step‚230 & lr-AdamW/pg1‚0.0002599274022031181æ,×t‚2Ð- _wandbruntime‚2206  _runtime‚2206)‚|Ù'ÑÓO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41314752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13658030 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167645618785792 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2381979957248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182998333500416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2373865692525 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41314754 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53723343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53723343 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167645618785792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95038097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180497576135168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500910503936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160939486219264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160939469179904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167646293150720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165263638828544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180497600075264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2500757365248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182998510579200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41314752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 % _timestamp‚1.7884071036443765e+09 _runtime‚2206.632089567 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24104006 trainer/global_step‚2849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2381979957248 _step‚231 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167646293150720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165263638828544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180497600075264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2500757365248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182998510579200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41314752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 % _timestamp‚1.7884071036443765e+09 _runtime‚2206.632089567 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24104006 trainer/global_step‚2849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2381979957248 _step‚231 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2381979957248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMo„…cLcnitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182998333500416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2373865692525 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41314754 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53723343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53723343 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167645618785792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95038097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0爫Ú2Ó- _wandbruntime‚2207  _runtime‚2207Á;eã&ÔÝM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165320009870336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13662957 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41328770 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2382799385088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2501776822784 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41328768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160994390349824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2383455760384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37775131 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183060833793024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53741823 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167702791252480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37774684 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2501606920192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53721760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163369072477521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160994373310464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167703465630720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2207.314127015 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2382799385088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2383455760384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2374027689737 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53721760 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13662512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24112174 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180559226872832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚2849 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165319991867392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95050528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41328770 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41328768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24112172 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2374682127697 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180559250812928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 % _timestamp‚1.7884071043263118e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165320009870336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167703465630720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165319991867392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944ë"pÀè&ÕâM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2207.314127015 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2382799385088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2383455760384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2374027689737 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53721760 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13662512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24112174 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180559226872832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚2849 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165319991867392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95050528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41328770 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41328768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24112172 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2374682127697 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180559250812928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 % _timestamp‚1.7884071043263118e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165320009870336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167703465630720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165319991867392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165320009870336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13662957 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41328770 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2382799385088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2501776822784 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41328768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160994390349824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2383455760384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37775131 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183060833793024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53741823 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167702791252480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37774684 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2501606920192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53721760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163369072477521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160994373310464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167703465630720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576è2&2Ö- _wandbruntime‚2207  _runtime‚2207²¿PÓ[×Õ epoch‚2 0loss/FBt1/assignment_loss‚0.6056551415887144 0loss/SRqqt1/detection_loss‚2.737598770651324 /loss/FBt1/detection_loss‚2.8607181310653687 % _timestamp‚1.7884071043277943e+09 0loss/FRt4/assignment_loss‚3.4250344010286553 /loss/FRt4/detection_loss‚2.5672861365384834 1loss/SRqqt4/detection_loss‚1.7238442654219956 0loss/FBt2/assignment_loss‚0.4310250586519639 1loss/FBt4/assignment_loss‚0.11121451531071216 -loss/FRt1/detection_loss‚2.85058089038905 1loss/SRqqt3/detection_loss‚1.1240406182862945 /loss/FBt4/detection_loss‚2.9300551295280455 _step‚233 /loss/FRt1/assignment_loss‚4.004947739488938 1loss/SRqqt2/detection_loss‚1.7998390207265287 _runtime‚2207.315135338 trainer/global_step‚2849 0loss/FRt2/assignment_loss‚3.7906769797915505 1loss/SRqqt2/assignment_loss‚2.060134564942502 2loss/SRqqt3/assignment_loss‚1.2834530717898476 2loss/SRqqt4/assignment_loss‚1.9746784097873247 /loss/FBt3/detection_loss‚0.6193221111533314 2loss/SRqqt1/assignment_loss‚2.4814658904897757 1loss/FBt3/assignment_loss‚0.22572499162531284 .loss/FRt2/detection_loss‚2.393580702089128 /loss/FRt3/detection_loss‚2.0561723339816798 &loss/total_loss‚48.152614189932464 /loss/FRt3/assignment_loss‚1.831719881814459 /loss/FBt2/detection_loss‚2.2638454337914786’o) `ØÚ _runtime‚2207.315135338 trainer/global_step‚2849 0loss/FRt2/assignment_loss‚3.7906769797915505 1loss/SRqqt2/assignment_loss‚2.060134564942502 2loss/SRqqt3/assignment_loss‚1.2834530717898476 2loss/SRqqt4/assignment_loss‚1.9746784097873247 /loss/FBt3/detection_loss‚0.6193221111533314 2loss/SRqqt1/assignment_loss‚2.4814658904897757 1loss/FBt3/assignment_loss‚0.22572499162531284 .loss/FRt2/detection_loss‚2.393580702089128 /loss/FRt3/detection_loss‚2.0561723339816798 &loss/total_loss‚48.152614189932464 /loss/FRt3/assignment_loss‚1.831719881814459 /loss/FBt2/detection_loss‚2.2638454337914786 epoch‚2 0loss/FBt1/assignment_loss‚0.6056551415887144 0loss/SRqqt1/detection_loss‚2.737598770651324 /loss/FBt1/detection_loss‚2.8607181310653687 % _timestamp‚1.7884071043277943e+09 0loss/FRt4/assignment_loss‚3.4250344010286553 /loss/FRt4/detection_loss‚2.5672861365384834 1loss/SRqqt4/detection_loss‚1.7238442654219956 0loss/FBt2/assignment_loss‚0.4310250586519639 1loss/FBt4/assignment_loss‚0.11121451531071216 -loss/FRt1/detection_loss‚2.85058089038905 1loss/SRqqt3/detection_loss‚1.1240406182862945 /loss/FBt4/detection_loss‚2.9300551295280455 _step‚233 /loss/FRt1/assignment_loss‚4.004947739488938 1loss/SRqqt2/detection_loss‚1.7998390207265287éIfÞˆ’Ù‚(:‡ ÍÚãÔ”ò gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÀtóSÚ‚(:È ÍÚãÔ¤ë  disk.sda2.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 12.52734375 disk.sda3.out‚ 105.2265625 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 12.52734375' disk.nvme0c0n1.out‚6134.96533203125 disk.sda.in‚0' proc.memory.availableMB‚ 476.4921875$ disk./.usageGB‚236.2455596923828 disk.sda.out‚ 105.2265625 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 12.52734375$ memory_percent‚97.09172248840332 disk.dm-1.out‚ 105.2265625 disk.drbd2265.out‚0 disk.sda3.in‚0 network.sent‚ 5.853059e+06* proc.memory.percent‚17.216205596923828 network.recv‚828228 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 11.8515625% disk.nvme0n1.out‚6134.96533203125" disk.dm-0.out‚6134.96533203125! proc.memory.rssMB‚ 2820.703125 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 cpu‚25.322683286468962 proc.cpu.threads‚17* disk./.usagePercent‚15.854637734183404 disk.drbd2266.in‚0 disk.sda1.out‚0²ÁáÛ‚(:’ ÜÚãÔ׶ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚129.949' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.989800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å×ÉèNÜ‚(:à ÜÚãÔÑÙÏ " disk.dm-0.out‚6138.25830078125 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 12.52734375% disk.nvme0n1.out‚6138.25830078125 disk.drbd2265.in‚0 disk.sda.in‚0$ memory_percent‚97.09014892578125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 12.52734375 disk.dm-1.in‚0 disk.dm-1.out‚ 105.8828125 disk.sda.out‚ 105.8828125* proc.memory.percent‚17.216205596923828 network.recv‚833597% disk./.usageGB‚236.24556350708008 disk.sda1.out‚0 disk.sda3.out‚ 105.8828125! disk.nvme0c0n1.in‚ 12.52734375 proc.cpu.threads‚17 network.sent‚ 5.960731e+06 disk.rbd0.out‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2820.703125 cpu‚25.320790940679498 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 11.9140625 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚6138.25830078125 disk.drbd1716.in‚0 disk.drbd1716.out‚0" proc.memory.availableMB‚476.75) disk./.usagePercent‚15.85463799019094 disk.drbd2080.in‚0 disk.sda2.out‚0QÍ-Ã2Ý- _wandbruntime‚2242  _runtime‚2242JwžÈÞ ' lr-AdamW/pg1‚0.00025900932216178156 ' lr-AdamW/pg2‚0.00025900932216178156 trainer/global_step‚2899 $ _timestamp‚1.788407139851748e+09 _runtime‚2242.839202956 _step‚234÷·³ÍßÇ ' lr-AdamW/pg1‚0.00025900932216178156 ' lr-AdamW/pg2‚0.00025900932216178156 trainer/global_step‚2899 $ _timestamp‚1.788407139851748e+09 _runtime‚2242.839202956 _step‚234ê[ÐÏ2à- _wandbruntime‚2242  _runtime‚2242ðÙÙzáÑO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54629358 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24512404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13886583 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2543990997504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38398989 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2899 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42015552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163684692747264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54649395 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170504930898944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42015554 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54629358 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168081308773888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13886135 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050304 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170504256534016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2414701031933 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183580136958464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚183580113018368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168081308773888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42015552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42015554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168081290770944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166099393779197 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2422965763072 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54649395 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38398539 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 6714771126 ÄÜ'âÖO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42015552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42015554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168081290770944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166099393779197 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2422965763072 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54649395 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38398539 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54629358 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24512404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13886583 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2543990997504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38398989 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2899 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42015552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163684692747264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54649395 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170504930898944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42015554 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54629358 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168081308773888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13886135 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050304 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170504256534016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2414701031933 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183580136958464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚183580113018368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168081308773888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928ë±')—2ã- _wandbruntime‚2243  _runtime‚224316‰­ã&äÝM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚2899 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168137661812736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54667875 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38411779 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24520574 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚236 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884071405259569e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24520572 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚2243.513898817 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163739579838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54667875 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42029568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170562103378944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38411332 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166155114344929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168137643809792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96697445 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2423785190912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96677380 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168137661812736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186186636567552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2414863029145 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2424441566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13891205 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872‘;ÖˆåâM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42029570 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54647812 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186186451113472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183641763756032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163739579838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54667875 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42029568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170562103378944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38411332 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166155114344929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168137643809792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96697445 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2423785190912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96677380 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168137661812736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186186636567552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2414863029145 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2424441566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13891205 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_poo9‹XY l.current‚ 868220928 trainer/global_step‚2899 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168137661812736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54667875 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38411779 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24520574 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚236 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884071405259569e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24520572 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚2243.513898817 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173362894282240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14115260 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168884764169493 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173363568647168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39036066 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186662673841664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2464607930880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170898960716288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173362894282240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14115260 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168884764169493 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173363568647168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39036066 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186662673841664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2464607930880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170898960716288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98304232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24928972 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14119376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55573864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.fre…«WÙCed‚189311951428096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55573864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2456352806513 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42730368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24928974 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39048348 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170955295752192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170955295752192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168940484735017 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166484786366464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42730370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173420741127168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14119846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2587650788864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98304232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168©wvíç&õáM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173420741127168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14119846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2587650788864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98304232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42730370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55593927 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39048820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186724324579328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98304232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24928972 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14119376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55573864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189311951428096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55573864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2456352806513 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42730368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24928974 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39048348 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170955295752192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170955295752192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168940484735017 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166484786366464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42730370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430ðPúÎ72ö- _wandbruntime‚2279  _runtime‚22792xÖj\÷Ö 2loss/SRqqt3/assignment_loss‚1.3544361835488907 0loss/FBt1/assignment_loss‚0.5249214164084859 /loss/FRt4/detection_loss‚2.3023166997092113 0loss/SRqqt3/detection_loss‚1.102669554261061 /loss/FBt3/detection_loss‚0.6865325253698025 _runtime‚2279.585620895 0loss/FRt3/assignment_loss‚2.1087866576891097 2loss/SRqqt2/assignment_loss‚1.9187512327493592 1loss/FBt4/assignment_loss‚0.33866668678820133 /loss/FBt2/detection_loss‚1.8306859185298283 &loss/total_loss‚48.440694138260945 .loss/FRt1/assignment_loss‚4.28888335161739 0loss/FBt2/assignment_loss‚0.6120909408976635 /loss/FBt1/detection_loss‚2.9876959323883057 .loss/FBt4/detection_loss‚3.773122638463974 .loss/FRt2/detection_loss‚2.287507804957303 /loss/FRt3/detection_loss‚1.7794250051180522 1loss/SRqqt2/detection_loss‚1.8804590884934773 trainer/global_step‚2949 _step‚241 0loss/FRt4/assignment_loss‚3.4341736958951365 0loss/SRqqt1/detection_loss‚2.378914164920007 2loss/SRqqt4/assignment_loss‚1.5453263920628362 1loss/FBt3/assignment_loss‚0.21427000261949652 epoch‚2 % _timestamp‚1.7884071765982401e+09 1loss/SRqqt4/detection_loss‚1.7770308060226616 2loss/SRqqt1/assignment_loss‚2.4624322873930775 /loss/FRt1/detection_loss‚2.8382938272423215 0loss/FRt2/assignment_loss‚4.0133013251152905›ÎraøÛ 1loss/FBt3/assignment_loss‚0.21427000261949652 epoch‚2 % _timestamp‚1.7884071765982401e+09 1loss/SRqqt4/detection_loss‚1.7770308060226616 2loss/SRqqt1/assignment_loss‚2.4624322873930775 /loss/FRt1/detection_loss‚2.8382938272423215 0loss/FRt2/assignment_loss‚4.0133013251152905 2loss/SRqqt3/assignment_loss‚1.3544361835488907 0loss/FBt1/assignment_loss‚0.5249214164084859 /loss/FRt4/detection_loss‚2.3023166997092113 0loss/SRqqt3/detection_loss‚1.102669554261061 /loss/FBt3/detection_loss‚0.6865325253698025 _runtime‚2279.585620895 0loss/FRt3/assignment_loss‚2.1087866576891097 2loss/SRqqt2/assignment_loss‚1.9187512327493592 1loss/FBt4/assignment_loss‚0.33866668678820133 /loss/FBt2/detection_loss‚1.8306859185298283 &loss/total_loss‚48.440694138260945 .loss/FRt1/assignment_loss‚4.28888335161739 0loss/FBt2/assignment_loss‚0.6120909408976635 /loss/FBt1/detection_loss‚2.9876959323883057 .loss/FBt4/detection_loss‚3.773122638463974 .loss/FRt2/detection_loss‚2.287507804957303 /loss/FRt3/detection_loss‚1.7794250051180522 1loss/SRqqt2/detection_loss‚1.8804590884934773 trainer/global_step‚2949 _step‚241 0loss/FRt4/assignment_loss‚3.4341736958951365 0loss/SRqqt1/detection_loss‚2.378914164920007 2loss/SRqqt4/assignment_loss‚1.5453263920628362ñÁÒA’ù‚(:‡ ‰ÛãÔ˜µ¦ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥š½êTú‚(:É ‰ÛãÔâÝë * disk./.usagePercent‚15.943629793198344 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 12.1259765625" disk.dm-0.out‚7498.46533203125 disk.dm-1.out‚ 107.57421875! proc.memory.rssMB‚ 2820.703125 proc.cpu.threads‚17 network.sent‚ 6.081443e+06 network.recv‚848717 disk.nvme0c0n1.in‚ 15.9453125% disk.nvme0n1.out‚7498.46533203125* proc.memory.percent‚17.216205596923828 disk.sda3.out‚ 107.57421875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 15.9453125 disk.sda.out‚ 107.57421875' disk.nvme0c0n1.out‚7498.46533203125 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 15.9453125# memory_percent‚97.0921516418457& proc.memory.availableMB‚ 476.421875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd1716.out‚0 cpu‚25.31547690002581 disk.rbd1.in‚0% disk./.usageGB‚237.57160568237305 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0¿i÷’û‚(:‡ ˜ÛãÔׇ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.066' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëô_ÊSü‚(:È ˜ÛãÔÇÖ¡  disk.drbd2265.out‚0 cpu‚25.313750725901798* disk./.usagePercent‚15.943836903293542 disk.drbd1716.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.rbd1.out‚ 12.1884765625 disk.nvme0n1.in‚ 16.6953125 disk.sda.out‚ 108.0390625$ memory_percent‚97.08948135375977! proc.memory.rssMB‚ 2820.703125 network.sent‚ 6.189001e+06 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 476.859375 network.recv‚854016 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚7504.27783203125 disk.dm-0.in‚ 16.6953125 disk.dm-1.out‚ 108.0390625 disk.sda1.in‚0* proc.memory.percent‚17.216205596923828" disk.dm-0.out‚7504.27783203125 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚7504.27783203125% disk./.usageGB‚237.57469177246094 disk.sda3.out‚ 108.0390625Ô–úD’ý‚(:‡ §ÛãÔœ‘¡ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 (î3Gþ‚(:¼ §ÛãÔ§ˆ  disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125" disk.dm-0.out‚7504.66455078125 disk.sda.out‚108.625 disk.drbd2080.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 16.6953125 disk.drbd2265.out‚0 disk.sda3.out‚108.625( proc.memory.availableMB‚ 477.05078125* proc.memory.percent‚17.216205596923828% disk./.usageGB‚237.57413864135742 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 16.6953125' disk.nvme0c0n1.out‚7504.66455078125! proc.memory.rssMB‚ 2820.703125 cpu‚25.31206075506222 network.recv‚858178 disk.rbd1.out‚ 12.1884765625 disk.dm-1.in‚0 disk.dm-1.out‚108.625 disk.drbd2266.out‚0) disk./.usagePercent‚15.94379978220108 disk.rbd1.in‚0 disk.rbd0.out‚0 network.sent‚ 6.194177e+06 disk.sda2.in‚0 disk.drbd1716.in‚0% disk.nvme0n1.out‚7504.66455078125 disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0$ memory_percent‚97.08831310272217*e|u2ÿ- _wandbruntime‚2315  _runtime‚2315Ÿ}ÿ\Ȁ _step‚242 ' lr-AdamW/pg1‚0.00025709741005846485 ' lr-AdamW/pg2‚0.00025709741005846485 trainer/global_step‚2999 $ _timestamp‚1.788407212261248e+09 _runtime‚2315.248646284줄ÍÇ $ _timestamp‚1.788407212261248e+09 _runtime‚2315.248646284 _step‚242 ' lr-AdamW/pg1‚0.00025709741005846485 ' lr-AdamW/pg2‚0.00025709741005846485 trainer/global_step‚2999ò=½?02‚- _wandbruntime‚2315  _runtime‚2315Ox¡»ÚƒÒO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171671477514013 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173716612658688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173716594655744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173716612658688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2496371710749 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14343082 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176222206395392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚2999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192374887847424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171670806036901 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56481462 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56481462 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2495717272997 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2505593736704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884072122619748e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176221532030464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99898614 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99918653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25329204 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189745186784768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173716594655744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39672715 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43417152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25329206 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56501499 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43417154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629701062656 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169175088763904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176222206395392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2504937374720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176221532030464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14343509 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173716594655744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39672715 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43417152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25329206 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56501499 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43417154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629701062656 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169175088763904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176222206395392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2504937374720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176221532030464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14343509 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176221532030464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99898614 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99918653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25329204 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189745186784768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872óÆ|U*2…- _wandbruntime‚2315  _runtime‚2315—¨ª±ã&†ÝM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169229992894464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173772947694592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56499916 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506413177856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14347747 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171726526602425 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43431170 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43431168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 _runtime‚2315.937368559 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497188145921 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2506413177856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169230009933824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56499916 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173772965697536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39685554 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171727198079745 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56519979 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176279378875392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39685119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173772965697536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43431170 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173772947694592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176278704497152 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25337374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884072129493096e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192437589828608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189806861462528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2630562657792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚244 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189806837522432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25337372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2496533707961 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505756802560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517®Tuè&‡âM trainer/global_step‚2999 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99951149 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192437400180224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56499916 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173772965697536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39685554 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171727198079745 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56519979 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176279378875392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39685119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173772965697536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43431170 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173772947694592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176278704497152 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25337374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884072129493096e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192437589828608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189806861462528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2630562657792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚244 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189806837522432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25337372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2496533707961 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505756802560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169229992894464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173772947694592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56499916 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506413177856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14347747 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171726526602425 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43431170 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43431168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 _runtime‚2315.937368559 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497188145921 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2506413177856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169230009933824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2546579542528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179080844143616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57427551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57407514 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101525466 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2672773139968 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44117954 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179080169778688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44117952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176534246598144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2546579542528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40309104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2545923180544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44117954 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174456847904309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44117952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176534264601088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2545923180544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14571950 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14571500 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2537207050157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176534264601088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179080844143616 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179080169778688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884072482404876e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192827747608064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2672936764416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171920312331264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174457519381421 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25737604 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2536552612405 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195500684372480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195500496807936µó’qâ'”ÜO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192827747608064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2672936764416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171920312331264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174457519381421 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25737604 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2536552612405 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195500684372480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195500496807936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2546579542528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179080844143616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57427551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57407514 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101525466 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2672773139968 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44117954 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179080169778688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44117952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176534246598144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2546579542528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40309104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2545923180544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44117954 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174456847904309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44117952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176534264601088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2545923180544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14571950 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14571500 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2537207050157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176534264601088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179080844143616 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179080169778688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884072482404876e+09÷DNÿ2•- _wandbruntime‚2351  _runtime‚23513›! –áM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176590599636992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195563190149632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 trainer/global_step‚3049 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2538023485329 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192889374405632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25745774 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2673621901312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44131968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2537369047369 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171975199422464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101557936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179138016623616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57425968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14576018 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57425968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195562996306944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179138016623616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192889398345728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2673791803904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176590617639936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176590617639936 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2547398983680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44131968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40321790 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57446031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14576467 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚2351.905916861 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25745772 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 trainer/global_step‚3049 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2538023485329 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192889374405632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25745774 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2673621901312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44131968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2537369047369 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171975199422464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101557936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179138016623616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57425968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14576018 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57425968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195562996306944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179138016623616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192889398345728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2673791803904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176590617639936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176590617639936 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2547398983680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44131968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40321790 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57446031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14576467 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚2351.905916861 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25745772 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2716128958464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195910284491264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚2387.317971433 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58333566 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181938807526912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884072843303173e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40945136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103152318 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58333566 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58353603 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198626230079488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715969528320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577387951813 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177242889771717 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44818754 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181939481891840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44818752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174665518859264 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181938807526912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26146006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14799130 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44818752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181939481891840 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179351916543488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44818754 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179351916543488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626413449728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179351898540544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 _step‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2586908986368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181938807526912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884072843303173e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40945136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103152318 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58333566 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58353603 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198626230079488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715969528320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577387951813 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177242889771717 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44818754 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181939481891840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44818752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174665518859264 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181938807526912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26146006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14799130 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44818752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181939481891840 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179351916543488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44818754 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179351916543488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626413449728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179351898540544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 _step‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2586908986368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2716128958464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195910284491264û«Ñ¦2§- _wandbruntime‚2388  _runtime‚23887ÝPæ&¨àM PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40957449 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103184788 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2578858824737 _runtime‚2388.003950687 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44832770 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40957906 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179408269582336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14803277 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103184788 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177298610337241 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716811097088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181996654371840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44832770 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179408251579392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198688722385920 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26154174 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177299281814561 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2578204386777 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26154172 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 _step‚252 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588384789504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103204853 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195971935228928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14803732 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181995979993600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44832768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174720422989824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179408251579392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198688920422912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0•‰öë&©åM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195971935228928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14803732 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181995979993600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44832768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174720422989824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179408251579392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198688920422912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40957449 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103184788 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2578858824737 _runtime‚2388.003950687 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44832770 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40957906 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179408269582336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14803277 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103184788 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177298610337241 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716811097088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181996654371840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44832770 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179408251579392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198688722385920 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26154174 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177299281814561 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2578204386777 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26154172 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 _step‚252 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588384789504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103204853ü)%ÜØ2ª- _wandbruntime‚2388  _runtime‚2388–6CW«Ñ 0loss/FRt4/assignment_loss‚3.4473818122072424 1loss/SRqqt2/assignment_loss‚1.981587853756818 /loss/FBt4/detection_loss‚4.1379163563251495 0loss/SRqqt1/detection_loss‚2.206110790371895 1loss/SRqqt2/detection_loss‚1.9759010531685568 1loss/SRqqt3/detection_loss‚1.0555303355350214 2loss/SRqqt3/assignment_loss‚1.2574261445278192 trainer/global_step‚3099 $ _timestamp‚1.788407285017966e+09 0loss/FRt3/assignment_loss‚1.6177917955250576 0loss/FBt1/assignment_loss‚0.5029078767531447 /loss/FBt2/detection_loss‚2.1339891508221625 /loss/FBt3/detection_loss‚0.5212363667735698 _runtime‚2388.005348804 _step‚253 /loss/FRt2/assignment_loss‚3.551412146538496 /loss/FRt1/assignment_loss‚4.350469307465986 /loss/FRt2/detection_loss‚2.9294989379122853 /loss/FRt3/detection_loss‚1.7825901363173435 %loss/total_loss‚48.78138536565584 1loss/SRqqt1/assignment_loss‚2.450301495898101 2loss/SRqqt4/assignment_loss‚1.8333352813497186 1loss/FBt3/assignment_loss‚0.21610945269862958 /loss/FBt4/assignment_loss‚0.514202781021595 1loss/SRqqt4/detection_loss‚1.8718987932162625 /loss/FBt1/detection_loss‚2.5984491341643863 0loss/FBt2/assignment_loss‚0.7215301364660263 .loss/FRt1/detection_loss‚2.905359873265931 .loss/FRt4/detection_loss‚2.218448353574631 epoch‚2­¼+X\¬Ö 2loss/SRqqt3/assignment_loss‚1.2574261445278192 trainer/global_step‚3099 $ _timestamp‚1.788407285017966e+09 0loss/FRt3/assignment_loss‚1.6177917955250576 0loss/FBt1/assignment_loss‚0.5029078767531447 /loss/FBt2/detection_loss‚2.1339891508221625 /loss/FBt3/detection_loss‚0.5212363667735698 _runtime‚2388.005348804 _step‚253 /loss/FRt2/assignment_loss‚3.551412146538496 /loss/FRt1/assignment_loss‚4.350469307465986 /loss/FRt2/detection_loss‚2.9294989379122853 /loss/FRt3/detection_loss‚1.7825901363173435 %loss/total_loss‚48.78138536565584 1loss/SRqqt1/assignment_loss‚2.450301495898101 2loss/SRqqt4/assignment_loss‚1.8333352813497186 1loss/FBt3/assignment_loss‚0.21610945269862958 /loss/FBt4/assignment_loss‚0.514202781021595 1loss/SRqqt4/detection_loss‚1.8718987932162625 /loss/FBt1/detection_loss‚2.5984491341643863 0loss/FBt2/assignment_loss‚0.7215301364660263 .loss/FRt1/detection_loss‚2.905359873265931 .loss/FRt4/detection_loss‚2.218448353574631 epoch‚2 0loss/FRt4/assignment_loss‚3.4473818122072424 1loss/SRqqt2/assignment_loss‚1.981587853756818 /loss/FBt4/detection_loss‚4.1379163563251495 0loss/SRqqt1/detection_loss‚2.206110790371895 1loss/SRqqt2/detection_loss‚1.9759010531685568 1loss/SRqqt3/detection_loss‚1.0555303355350214ým­­‚(:’ ÜãÔ¡ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.631200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž³“ ®‚(:Æ ÜãÔ¨õ¾  disk.rbd0.out‚0 disk.nvme0n1.in‚ 16.6953125" disk.dm-0.out‚7507.43603515625* proc.memory.percent‚17.216205596923828* °rŒ±disk./.usagePercent‚15.943817702728474 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚0 network.recv‚887100 disk.dm-0.in‚ 16.6953125 disk.drbd2265.in‚0% disk./.usageGB‚237.57440567016602 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 12.619140625 cpu‚25.302162987672627 disk.drbd2080.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚7507.43603515625 disk.dm-1.in‚0 disk.dm-1.out‚ 111.640625 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2820.703125 disk.nvme0c0n1.in‚ 16.6953125 network.sent‚ 6.434538e+06 disk.sda.out‚ 111.640625$ memory_percent‚97.09146022796631( proc.memory.availableMB‚ 476.53515625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚7507.43603515625 disk.sda.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 111.640625 proc.cpu.threads‚17A`ˆ’¯‚(:‡ ÜãÔÞË gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÉW°‚(:Ì ÜãÔøä¨  disk.rbd1.out‚ 12.681640625 disk.drbd2080.out‚0 disk.sda.out‚ 112.08984375$ memory_percent‚97.09322452545166! proc.memory.rssMB‚ 2820.703125* proc.memory.percent‚17.216205596923828 disk.drbd1716.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.drbd2266.out‚0 network.sent‚ 6.542218e+06* disk./.usagePercent‚15.943817958736009 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 16.6953125 disk.dm-0.in‚ 16.6953125 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 476.24609375% disk.nvme0n1.out‚7508.54345703125" disk.dm-0.out‚7508.54345703125 disk.dm-1.out‚ 112.08984375 disk.sda1.in‚0 network.recv‚892719% disk./.usageGB‚237.57440948486328 disk.rbd1.in‚0 cpu‚25.300594790227187 proc.cpu.threads‚17 disk.sda3.out‚ 112.08984375 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚7508.54345703125 disk.drbd2265.out‚0ŠiòU2±- _wandbruntime‚2423  _runtime‚2423_:¨ùȲ _step‚254 ' lr-AdamW/pg1‚0.00025404318915720874 ' lr-AdamW/pg2‚0.00025404318915720874 trainer/global_step‚3149 $ _timestamp‚1.788407320401111e+09 _runtime‚2423.388551437nCÛÍ³Ç ' lr-AdamW/pg1‚0.00025404318915720874 ' lr-AdamW/pg2‚0.00025404318915720874 trainer/global_step‚3149 $ _timestamp‚1.788407320401111e+09 _runtime‚2423.388551437 _step‚254þðYŠ2´- _wandbruntime‚2423  _runtime‚2423/¹G¢Ý'µ×O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180028931639125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177410708347904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2618877728973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184797445275136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2627894792192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177410725387264 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45519552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2627894792192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59259618 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2628551154176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182169568485888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2628551154176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26554404 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198992821374464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45519552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚2423.389529662 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2759255388160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45519554 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182169568485888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177410708347904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2618877728973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184797445275136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2627894792192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177410725387264 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45519552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2627894792192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59259618 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2628551154176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182169568485888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2628551154176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26554404 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198992821374464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45519552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚2423.389529662 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2759255388160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45519554 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182169568485888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104831705 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45533568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104811640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45533568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41594327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15031303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201814402153472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2760117592576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180084652204649 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2759953981440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚184854617741824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59298135 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199054448172032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180085323681969 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚3149 _runtime‚2424.080195877 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45533570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184855292120064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41593875 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26562572 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2619694164145 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2629370595328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59278072 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182225921524736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182225921524736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15031753 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021jâKQì&¹æM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚3149 _runtime‚2424.080195877 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45533570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184855292120064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41593875 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26562572 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2619694164145 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2629370595328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59278072 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182225921524736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182225921524736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15031753 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚256 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚184855292120064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177465629517824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182225903521792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104831705 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45533568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104811640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45533568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41594327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15031303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201814402153472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2760117592576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180084652204649 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2759953981440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚184854617741824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59298135 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199054448172032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180085323681969€Ð@¬Á2º- _wandbruntime‚2424  _runtime‚2424‚è€IW»Ñ /loss/FRt4/assignment_loss‚3.466650603633178 /loss/FRt3/detection_loss‚1.9749139013725359 /loss/FBt3/detection_loss‚0.7723837036039465 1loss/FBt2/assignment_loss‚0.29824562690087425 /loss/FRt2/detection_loss‚2.1007772150792574 .loss/FBt2/detection_loss‚2.399437572274889 $loss/total_loss‚48.3548905224571 2loss/SRqqt3/assignment_loss‚1.2343374615454155 1loss/FBt3/assignment_loss‚0.19030549535489472 /loss/FRt1/detection_loss‚2.9557171426713467 1loss/SRqqt4/detection_loss‚2.0528336509510323 .loss/FBt1/detection_loss‚3.746092826128006 epoch‚2 _runtime‚2424.081688633 0loss/FRt2/assignment_loss‚3.3195573802579914 2loss/SRqqt2/assignment_loss‚1.5130976920149155 1loss/FBt4/assignment_loss‚0.33151763677597046 /loss/FRt4/detection_loss‚2.5182193501999506 0loss/FRt3/assignment_loss‚1.7671269194499866 0loss/FBt1/assignment_loss‚0.2765362511078517 0loss/SRqqt3/detection_loss‚1.148857334655696 $ _timestamp‚1.788407321094314e+09 2loss/SRqqt1/assignment_loss‚2.2109177647492824 1loss/SRqqt1/detection_loss‚2.3442905147870383 .loss/FBt4/detection_loss‚4.198100566864014 /loss/FRt1/assignment_loss‚4.113653130829334 trainer/global_step‚3149 _step‚257 0loss/SRqqt4/assignment_loss‚1.52002897127359 0loss/SRqqt2/detection_loss‚1.901291809976101¿@\¼Ö 0loss/FRt3/assignment_loss‚1.7671269194499866 0loss/FBt1/assignment_loss‚0.2765362511078517 0loss/SRqqt3/detection_loss‚1.148857334655696 $ _timestamp‚1.788407321094314e+09 2loss/SRqqt1/assignment_loss‚2.2109177647492824 1loss/SRqqt1/detection_loss‚2.3442905147870383 .loss/FBt4/detection_loss‚4.198100566864014 /loss/FRt1/assignment_loss‚4.113653130829334 trainer/global_step‚3149 _step‚257 0loss/SRqqt4/assignment_loss‚1.52002897127359 0loss/SRqqt2/detection_loss‚1.901291809976101 /loss/FRt4/assignment_loss‚3.466650603633178 /loss/FRt3/detection_loss‚1.9749139013725359 /loss/FBt3/detection_loss‚0.7723837036039465 1loss/FBt2/assignment_loss‚0.29824562690087425 /loss/FRt2/detection_loss‚2.1007772150792574 .loss/FBt2/detection_loss‚2.399437572274889 $loss/total_loss‚48.3548905224571 2loss/SRqqt3/assignment_loss‚1.2343374615454155 1loss/FBt3/assignment_loss‚0.19030549535489472 /loss/FRt1/detection_loss‚2.9557171426713467 1loss/SRqqt4/detection_loss‚2.0528336509510323 .loss/FBt1/detection_loss‚3.746092826128006 epoch‚2 _runtime‚2424.081688633 0loss/FRt2/assignment_loss‚3.3195573802579914 2loss/SRqqt2/assignment_loss‚1.5130976920149155 1loss/FBt4/assignment_loss‚0.33151763677597046 /loss/FRt4/detection_loss‚2.51821935019995061Äì]œ½‚(:‘ ŸÜãÔ·†Œ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.858' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.77160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A|ÆS¾‚(:È ŸÜãÔ¦¡Í  disk.sda2.in‚0 cpu‚25.29897750102555 disk.rbd1.out‚ 12.681640625! disk.dm-0.out‚7509.0244140625 disk.dm-1.out‚ 112.70703125 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚ 16.6953125* proc.memory.percent‚17.216205596923828$ disk.nvme0n1.out‚7509.0244140625 disk.drbd2266.in‚0 disk.sda2.out‚0 network.recv‚896842 disk.drbd1716.in‚0 disk.dm-0.in‚ 16.6953125 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0$ memory_percent‚97.09198474884033% disk./.usageGB‚237.57439041137695 disk.dm-1.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 476.44921875 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚7509.0244140625* disk./.usagePercent‚15.943816678698338 disk.nvme0n1.in‚ 16.6953125 disk.drbd2265.in‚0 disk.sda.out‚ 112.70703125! proc.memory.rssMB‚ 2820.703125 network.sent‚ 6.548218e+06 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 112.70703125 disk.rbd0.out‚0 disk.sda1.out‚0„ýü¼‘¿‚(:† ®ÜãÔ¹•õ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.085' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.017 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ߥ“PÀ‚(:Å ®ÜãÔر  disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 16.6953125 disk.dm-0.in‚ 16.6953125* proc.memory.percent‚17.216205596923828 cpu‚25.29743076940741 network.sent‚ 6.656706e+06 disk.drbd2080.out‚0! disk.dm-0.out‚7509.6494140625 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚7509.6494140625$ memory_percent‚97.09205627441406! proc.memory.rssMB‚ 2820.703125* disk./.usagePercent‚15.943832295157929 disk.sda.out‚ 113.24609375 disk.sda3.out‚ 113.24609375 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda1.in‚0 network.recv‚902958% disk./.usageGB‚237.57462310791016 disk.dm-1.out‚ 113.24609375 disk.sda.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 12.8076171875 disk.nvme0c0n1.in‚ 16.6953125& disk.nvme0c0n1.out‚7509.6494140625$ proc.memory.availableMB‚476.4375Ä)I¹2Á- _wandbruntime‚2459  _runtime‚2459Ë˧ÉÂà trainer/global_step‚3199 % _timestamp‚1.7884073562900798e+09 _runtime‚2459.277505842 _step‚258 ' lr-AdamW/pg1‚0.00025297626496565984 ' lr-AdamW/pg2‚0.00025297626496565984½æänÎÃÈ % _timestamp‚1.7884073562900798e+09 _runtime‚2459.277505842 _step‚258 ' lr-AdamW/pg1‚0.00025297626496565984 ' lr-AdamW/pg2‚0.00025297626496565984 trainer/global_step‚3199‚X°âf2Ä- _wandbruntime‚2459  _runtime‚2459ý\ŠæÝ'Å×O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182814973506533 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2659713068381 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46220354 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46220352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180155931915264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180155914875904 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46220352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187656083023360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 trainer/global_step‚3199 % _timestamp‚1.7884073562909317e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 _step‚259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60205707 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2669536960000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60185670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15254366 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182815644983645 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668880598016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26962806 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204877526140928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187656757388288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184987220428288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202075334317568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2802370127872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184987202425344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46220354 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15254826 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187656757388288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184987220428288 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187656083023360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2659058630629 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60205707 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384ßT§ÈfÆÜO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60205707 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182814973506533 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2659713068381 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46220354 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46220352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180155931915264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180155914875904 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46220352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187656083023360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 trainer/global_step‚3199 % _timestamp‚1.7884073562909317e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 _step‚259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60205707 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2669536960000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60185670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15254366 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182815644983645 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668880598016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26962806 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204877526140928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187656757388288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184987220428288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202075334317568 cQDeviceStatsMonitor.onjI`=|_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2802370127872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184987202425344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46220354 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15254826 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187656757388288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184987220428288 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187656083023360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2659058630629ƒ,Ç{,2Ç- _wandbruntime‚2459  _runtime‚2459ª'ébæ&ÈàM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185043573467136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180210819006464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46234368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182871365549377 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42230278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26970972 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46234370 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2669700025856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46234368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185043555464192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204940223152640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670356401152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46234370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60224187 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42229839p×ljë&ÉåM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2803048449024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202136985055232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182870694072057 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 $ _timestamp‚1.788407356965335e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60204124 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42230278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26970972 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46234370 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2669700025856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46234368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185043555464192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204940223152640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670356401152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46234370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60224187 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42229839 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185043573467136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180210819006464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46234368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182871365549377 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F*LЂ(:Á ÌÜãÔ×ÓÆ  disk.drbd2080.in‚0 disk.sda2.in‚0 disk.sda.out‚ 115.171875 disk.sda3.out‚ 115.171875 disk.rbd0.out‚0* proc.memory.percent‚17.216205596923828* disk./.usagePercent‚15.854771370116277 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.drbd2080.out‚0# memory_percent‚97.0952033996582 cpu‚25.294396907187622 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 16.6953125 disk.dm-1.in‚0% disk./.usageGB‚236.24755096435547& disk.nvme0c0n1.out‚7511.0224609375 disk.rbd1.out‚ 12.9560546875 disk.dm-1.out‚ 115.171875 disk.sda.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 475.921875$ disk.nvme0n1.out‚7511.0224609375 network.recv‚912972 disk.drbd1716.in‚0! disk.dm-0.out‚7511.0224609375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.dm-0.in‚ 16.6953125! proc.memory.rssMB‚ 2820.703125 network.sent‚ 6.770674e+06 disk.sda1.out‚0öxý’Ñ‚(:‡ ÛÜãÔîçª gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.487' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡xB[TÒ‚(:É ÛÜãÔ›éÉ  disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 12.9560546875 cpu‚25.29283928692654 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd1716.in‚0( proc.memory.availableMB‚ 476.17578125* disk./.usagePercent‚15.854771370116277 disk.dm-0.in‚ 16.6953125 disk.dm-1.out‚ 115.8359375 disk.drbd2080.out‚0$ memory_percent‚97.09365367889404* proc.memory.percent‚17.216205596923828 network.recv‚916970% disk.nvme0n1.out‚7511.40966796875 disk.sda.in‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2820.703125 disk.sda3.out‚ 115.8359375 disk.drbd2266.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚7511.40966796875" disk.dm-0.out‚7511.40966796875 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 6.776748e+06% disk./.usageGB‚236.24755096435547 disk.drbd2265.out‚0 disk.sda.out‚ 115.8359375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 16.6953125 disk.sda1.in‚0ßž ¥2Ó- _wandbruntime‚2495  _runtime‚2495ðÛ ÇÔÁ & lr-AdamW/pg2‚0.0002518852818465187 trainer/global_step‚3249 % _timestamp‚1.7884073924797537e+09 _runtime‚2495.467200681 _step‚262 & lr-AdamW/pg1‚0.0002518852818465187ë°å=ÌÕÆ & lr-AdamW/pg1‚0.0002518852818465187 & lr-AdamW/pg2‚0.0002518852818465187 trainer/global_step‚3249 % _timestamp‚1.7884073924797537e+09 _runtime‚2495.467200681 _step‚262†C¢z2Ö- _wandbruntime‚2495  _runtime‚2495˜”à«r××O KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_trainÍÙKÐk'_batch_start/inactive_split_bytes.small_pool.freed‚ 2845338584064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205157871200768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3249 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884073924804866e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187804854367744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61111722 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61131759 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208003391057920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190514720771584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42852860 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108052913 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208003209784832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46921152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190515395136512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42852411M~­lâ'ØÜO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46921152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190515395136512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42852411 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2845338584064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205157871200768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3249 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884073924804866e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187804854367744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61111722 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61131759 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208003391057920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190514720771584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42852860 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108052913 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208003209784832‡Ö·÷2Ù- _wandbruntime‚2496  _runtime‚2496ÌW¬ýç&ÚáM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 _runtime‚2496.142277084 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42865572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187861225409536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2710685831680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15485747 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚3249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46935168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177065472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187861207406592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190572567616512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185657407416785 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182956025534464 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185656735939465 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46935168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884073931544616e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61130176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208065736557056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153125376 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42865119 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27379374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2846214618624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187861225409536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187861207406592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46935170 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61150239 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2701364842961 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27379372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182956042573824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108085409 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2710685831680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15486198 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2711342206976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61130176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320B6ñÃw ÛæM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46935168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177065472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187861207406592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190572567616512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185657407416785 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182956025534464 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185656735939465 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on-E˜u_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46935168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884073931544616e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61130176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208065736557056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153125376 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42865119 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27379374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2846214618624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187861225409536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187861207406592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46935170 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61150239 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2701364842961 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27379372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182956042573824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108085409 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2710685831680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15486198 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2711342206976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61130176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 _runtime‚2496.142277084 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42865572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187861225409536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2710685831680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15485747 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚3249ˆJ.$(2Ü- _wandbruntime‚2496  _runtime‚24967lVÝÐ 1loss/SRqqt4/detection_loss‚1.8355582099821832 .loss/FRt2/detection_loss‚2.223296898106734 /loss/FRt1/assignment_loss‚4.240980540552447 /loss/FRt3/assignment_loss‚1.849196249583982 0loss/FRt4/assignment_loss‚3.3696434918571923 1loss/FBt4/assignment_loss‚0.07840008996427059 -loss/FRt4/detection_loss‚2.14139753872273 epoch‚2 1loss/FBt3/assignment_loss‚0.20553266306069679 1loss/SRqqt2/detection_loss‚1.9821957051753998 /loss/FBt3/detection_loss‚0.7415344821906183 /loss/FRt1/detection_loss‚3.0007993482774302 1loss/SRqqt3/detection_loss‚1.0038068948047503 /loss/FBt2/detection_loss‚2.7596403777599336 %loss/total_loss‚49.42632895688925 trainer/global_step‚3249 /loss/FBt2/assignment_loss‚0.492145784199235 _runtime‚2496.143449506 /loss/FBt1/detection_loss‚3.5711925506591795 _step‚265 2loss/SRqqt1/assignment_loss‚2.1994290947914124 2loss/SRqqt4/assignment_loss‚1.7821503811412387 0loss/FBt1/assignment_loss‚0.5611553817987442 -loss/FBt4/detection_loss‚4.37877858877182 $ _timestamp‚1.788407393156081e+09 /loss/FRt2/assignment_loss‚3.589719952394565 2loss/SRqqt2/assignment_loss‚2.0112914002322135 1loss/SRqqt3/assignment_loss‚1.092510366386601 /loss/FRt3/detection_loss‚1.7431892728046425 1loss/SRqqt1/detection_loss‚2.5727836936712265P[ÞÕ 2loss/SRqqt1/assignment_loss‚2.1994290947914124 2loss/SRqqt4/assignment_loss‚1.7821503811412387 0loss/FBt1/assignment_loss‚0.5611553817987442 -loss/FBt4/detection_loss‚4.37877858877182 $ _timestamp‚1.788407393156081e+09 /loss/FRt2/assignment_loss‚3.589719952394565 2loss/SRqqt2/assignment_loss‚2.0112914002322135 1loss/SRqqt3/assignment_loss‚1.092510366386601 /loss/FRt3/detection_loss‚1.7431892728046425 1loss/SRqqt1/detection_loss‚2.5727836936712265 1loss/SRqqt4/detection_loss‚1.8355582099821832 .loss/FRt2/detection_loss‚2.223296898106734 /loss/FRt1/assignment_loss‚4.240980540552447 /loss/FRt3/assignment_loss‚1.849196249583982 0loss/FRt4/assignment_loss‚3.3696434918571923 1loss/FBt4/assignment_loss‚0.07840008996427059 -loss/FRt4/detection_loss‚2.14139753872273 epoch‚2 1loss/FBt3/assignment_loss‚0.20553266306069679 1loss/SRqqt2/detection_loss‚1.9821957051753998 /loss/FBt3/detection_loss‚0.7415344821906183 /loss/FRt1/detection_loss‚3.0007993482774302 1loss/SRqqt3/detection_loss‚1.0038068948047503 /loss/FBt2/detection_loss‚2.7596403777599336 %loss/total_loss‚49.42632895688925 trainer/global_step‚3249 /loss/FBt2/assignment_loss‚0.492145784199235 _runtime‚2496.143449506 /loss/FBt1/detection_loss‚3.5711925506591795 _step‚265‰Í° ’ß‚(:‡ êÜãÔÿ§ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.387' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8774 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"®MÙOà‚(:Ä êÜãÔò“  disk.nvme0n1.in‚ 16.6953125 disk.dm-0.out‚7514.392578125 disk.sda3.in‚0% disk.nvme0c0n1.out‚7514.392578125* proc.memory.percent‚17.216205596923828 network.sent‚ 6.782545e+06 disk.drbd2080.in‚0 disk.sda1.out‚0 cpu‚25.291387596980186 disk.dm-1.out‚ 116.35546875# disk.nvme0n1.out‚7514.392578125 disk.drbd2265.out‚0 disk.sda1.in‚0* disk./.usagePercent‚15.854820779570385 disk.sda.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0% disk./.usageGB‚236.24828720092773 disk.drbd1716.in‚0 disk.sda.out‚ 116.35546875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 16.6953125$ memory_percent‚97.09258079528809 network.recv‚921285 disk.drbd1716.out‚0 disk.sda3.out‚ 116.35546875 disk.drbd2266.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 476.3515625 disk.rbd1.in‚0 disk.rbd1.out‚ 13.08203125 disk.dm-0.in‚ 16.6953125 disk.drbd2265.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2820.703125 proc.cpu.threads‚17 disk.dm-1.in‚0rQ\Sá‚(:’ ùÜãÔ¨¤³ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.215' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.443000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 4,Lâ‚(:Á ùÜãÔÖâô  disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0! proc.memory.rssMB‚ 2820.703125 proc.cpu.threads‚17 network.sent‚ 6.891243e+06 disk.rbd0.in‚0 disk.dm-1.in‚0 cpu‚25.289969816987536$ disk./.usageGB‚236.2482681274414 disk.rbd0.out‚0 disk.dm-1.out‚ 116.9921875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 116.9921875 disk.drbd2266.in‚0 network.recv‚927547 disk.sda1.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0% disk.nvme0c0n1.out‚7514.673828125 disk.rbd1.out‚ 13.16796875# disk.nvme0n1.out‚7514.673828125 disk.sda.out‚ 116.9921875 disk.drbd2080.out‚0* disk./.usagePercent‚15.854819499532717 disk.sda1.out‚0 disk.dm-0.out‚7514.673828125 disk.sda.in‚0( proc.memory.availableMB‚ 476.33984375 disk.nvme0n1.in‚ 16.6953125 disk.dm-0.in‚ 16.6953125$ memory_percent‚97.09265232086182* proc.memory.percent‚17.216205596923828 disk.nvme0c0n1.in‚ 16.6953125/= ³2ã- _wandbruntime‚2531  _runtime‚2531,u|'ÇäÁ & lr-AdamW/pg1‚0.0002507704671590322 & lr-AdamW/pg2‚0.0002507704671590322 trainer/global_step‚3299 % _timestamp‚1.7884074285374715e+09 _runtime‚2531.524968357 _step‚266ƒ["’ÌåÆ & lr-AdamW/pg1‚0.0002507704671590322 & lr-AdamW/pg2‚0.0002507704671590322 trainer/global_step‚3299 % _timestamp‚1.7884074285374715e+09 _runtime‚2531.524968357 _step‚266Š³¤ l2æ- _wandbruntime‚2531  _runtime‚2531i„ÚÒÝ'ç×O JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62037774 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193374032884736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109679765 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43488491 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62037774 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193373358519808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15708885 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190622524313088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27779606 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47621952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47621954 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190622506310144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2750852209664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27779604 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185646327931904 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193374032884736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43488060 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚3299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62057811 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211128639515136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2888386667008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208240432024064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚2531.525910424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2888231431168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185646344971264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15708456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109659726 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47621952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188387057241349 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2751508571648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2741383747197 % _timestamp‚1.7884074285382245e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190622524313088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2750852209664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47621954 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚267 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62057811 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188387728718461 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2740729309445 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193373358519808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208240408083968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190622506310144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15708456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109659726 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47621952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188387057241349 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2751508571648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2741383747197 % _timestamp‚1.7884074285382245e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190622524313088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2750852209664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47621954 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚267 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62057811 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188387728718461 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2740729309445 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193373358519808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208240408083968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190622506310144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62037774 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193374032884736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109679765 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43488491 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62037774‹&á2é- _wandbruntime‚2532  _runtime‚2532‘ó²Ëç&êáM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211191151848960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62056228 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185701249101824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2889093027328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884074292133203e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190678859348992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15712943 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109712261 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62076291 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62056228 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109712261 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188443449284193 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208302058821632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193431205364736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190678877351936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188442777806873 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185701232062464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47635968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193431205364736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193430530986496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 trainer/global_step‚3299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2752328012800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43501160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27787772 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2751671637504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109692196 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190678859348992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43500715 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27787774 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2889254541312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2742200182369 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2741545744409 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47635970 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928ßôtì&ëæM _runtime‚2532.201469748 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43500715 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27787774 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2889254541312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2742200182369 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2741545744409 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47635970 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211191151848960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62056228 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185701249101824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2889093027328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884074292133203e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190678859348992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15712943 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109712261 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62076291 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62056228 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109712261 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188443449284193 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208302058821632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193431205364736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190678877351936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188442777806873 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185701232062464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47635968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193431205364736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193430530986496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 trainer/global_step‚3299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2752328012800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43501160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27787772 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2751671637504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109692196 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190678859348992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E2ì- _wandbruntime‚2532  _runtime‚2532Þ¦úSíÍ .loss/FRt4/detection_loss‚2.583885138233503 0loss/SRqqt1/detection_loss‚2.306141956764109 2loss/SRqqt2/assignment_loss‚1.8107511041952031 /loss/FBt1/detection_loss‚3.1074735720952353 /loss/FBt2/detection_loss‚2.0254014174143475 2loss/SRqqt1/assignment_loss‚2.0334779497455147 2loss/SRqqt4/assignment_loss‚1.6552417528022219 1loss/SRqqt3/detection_loss‚0.9417963343687722 epoch‚2 2loss/SRqqt3/assignment_loss‚1.1110290588780505 0loss/FBt2/assignment_loss‚0.7798629065354362 0loss/FBt3/assignment_loss‚0.3152568673739512 1loss/SRqqt2/detection_loss‚1.9101993824754442 0loss/FBt4/assignment_loss‚0.6130779054015875 0loss/FRt3/assignment_loss‚1.8530883216390424 0loss/FBt1/assignment_loss‚0.8191895037889481 /loss/FRt3/detection_loss‚1.7258953003614557 # _timestamp‚1.78840742921514e+09 /loss/FRt1/assignment_loss‚3.386145245644354 0loss/SRqqt4/detection_loss‚1.872407984264471 /loss/FBt3/detection_loss‚0.6125557136518878 trainer/global_step‚3299 /loss/FRt1/detection_loss‚2.8919672543002712 -loss/FBt4/detection_loss‚3.49145884513855 %loss/total_loss‚47.36028129542733 _runtime‚2532.202504239 _step‚269 .loss/FRt2/assignment_loss‚3.63906676204581 /loss/FRt4/assignment_loss‚3.594445995795421 .loss/FRt2/detection_loss‚2.280465022513741?ÉXîÒ 0loss/FRt3/assignment_loss‚1.8530883216390424 0loss/FBt1/assignment_loss‚0.8191895037889481 /loss/FRt3/detection_loss‚1.7258953003614557 # _timestamp‚1.78840742921514e+09 /loss/FRt1/assignment_loss‚3.386145245644354 0loss/SRqqt4/detection_loss‚1.872407984264471 /loss/FBt3/detection_loss‚0.6125557136518878 trainer/global_step‚3299 /loss/FRt1/detection_loss‚2.8919672543002712 -loss/FBt4/detection_loss‚3.49145884513855 %loss/total_loss‚47.36028129542733 _runtime‚2532.202504239 _step‚269 .loss/FRt2/assignment_loss‚3.63906676204581 /loss/FRt4/assignment_loss‚3.594445995795421 .loss/FRt2/detection_loss‚2.280465022513741 .loss/FRt4/detection_loss‚2.583885138233503 0loss/SRqqt1/detection_loss‚2.306141956764109 2loss/SRqqt2/assignment_loss‚1.8107511041952031 /loss/FBt1/detection_loss‚3.1074735720952353 /loss/FBt2/detection_loss‚2.0254014174143475 2loss/SRqqt1/assignment_loss‚2.0334779497455147 2loss/SRqqt4/assignment_loss‚1.6552417528022219 1loss/SRqqt3/detection_loss‚0.9417963343687722 epoch‚2 2loss/SRqqt3/assignment_loss‚1.1110290588780505 0loss/FBt2/assignment_loss‚0.7798629065354362 0loss/FBt3/assignment_loss‚0.3152568673739512 1loss/SRqqt2/detection_loss‚1.9101993824754442 0loss/FBt4/assignment_loss‚0.6130779054015875ãP÷)’ï‚(:‡ ˆÝãÔ‹¼¼ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.679' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3358 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QHnâ@ð‚(:µ ˆÝãÔ¥º„  disk.rbd1.in‚0 disk.rbd1.out‚ 13.23046875 cpu‚25.288540863408066 disk.dm-1.out‚ 117.5234375 network.sent‚ 6.896481e+06 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 16.6953125$ proc.memory.availableMB‚476.3125* disk./.usagePercent‚15.854806443148467 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 16.6953125 disk.drbd2265.out‚0$ memory_percent‚97.09281921386719 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2265.in‚0! proc.memory.rssMB‚ 2820.703125 network.recv‚931251% disk./.usageGB‚236.24807357788086 disk.nvme0c0n1.in‚ 16.6953125 disk.nvme0n1.out‚ 7514.953125* proc.memory.percent‚17.216205596923828 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.out‚ 7514.953125 disk.drbd1716.out‚0 disk.sda.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0 disk.dm-0.out‚ 7514.953125 disk.sda.out‚ 117.5234375 proc.cpu.threads‚17 disk.sda3.out‚ 117.5234375 disk.drbd1716.in‚06–@Ññ‚(:† —ÝãÔîÍÇ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.325' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.065 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£m«¢Mò‚(: —ÝãÔâùž  disk.dm-0.out‚ 7515.23046875 disk.drbd2080.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 7515.23046875" disk.nvme0n1.out‚ 7515.23046875 disk.dm-1.in‚0 disk.drbd2265.out‚0 network.sent‚ 7.005057e+06 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 16.6953125( proc.memory.availableMB‚ 476.06640625* disk./.usagePercent‚15.854806699156002 disk.nvme0c0n1.in‚ 16.6953125 disk.rbd1.out‚ 13.23046875! proc.memory.rssMB‚ 2820.703125 cpu‚25.287167293426712 disk.drbd2080.out‚0 disk.sda.out‚ 118.07421875 disk.sda3.in‚0 disk.drbd2266.out‚0 network.recv‚937298% disk./.usageGB‚236.24807739257812 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda3.out‚ 118.07421875 disk.drbd2266.in‚0* proc.memory.percent‚17.216205596923828 disk.drbd2265.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 16.6953125 disk.dm-1.out‚ 118.07421875 proc.cpu.threads‚17 disk.drbd1716.in‚0$ memory_percent‚97.09432125091553.&º’ó‚(:‡ ¦ÝãԺІ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0váQô‚(:Æ ¦ÝãÔè¼ð  disk.drbd2080.in‚0 network.recv‚941379 disk.rbd1.in‚0 disk.dm-0.in‚ 16.6953125 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 cpu‚25.28580132447487 disk.rbd1.out‚13.45751953125 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.216205596923828 proc.cpu.threads‚17% disk./.usageGB‚236.24807739257812 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd1716.in‚0! disk.dm-0.out‚7516.5087890625 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 118.5703125 disk.sda.out‚ 118.5703125 disk.sda3.out‚ 118.5703125 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2820.703125$ disk.nvme0n1.out‚7516.5087890625 disk.drbd2265.in‚0 network.sent‚ 7.010894e+06& disk.nvme0c0n1.out‚7516.5087890625 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 16.6953125 disk.drbd2080.out‚0 disk.rbd0.in‚0$ memory_percent‚97.09286689758301' proc.memory.availableMB‚ 476.3046875* disk./.usagePercent‚15.854806699156002:D’#2õ- _wandbruntime‚2567  _runtime‚2567¢,šÅö¿ _step‚270 & lr-AdamW/pg1‚0.0002496320532289102 & lr-AdamW/pg2‚0.0002496320532289102 trainer/global_step‚3349 $ _timestamp‚1.788407464759728e+09 _runtime‚ 2567.74719948_ùÇbÊ÷Ä trainer/global_step‚3349 $ _timestamp‚1.788407464759728e+09 _runtime‚ 2567.74719948 _step‚270 & lr-AdamW/pg1‚0.0002496320532289102 & lr-AdamW/pg2‚0.0002496320532289102Ž¸°Þ]2ø- _wandbruntime‚2567  _runtime‚2567u þà ù×O F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48322754 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28188004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48322752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15937488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193440176255488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15937046 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196231996268032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62983863 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44125050 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44125494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111286578 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111306617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batchò ;èý_start/requested_bytes.large_pool.freed‚188391534459904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3349 _runtime‚2567.748178787 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193440176255488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2791838015488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2931310018560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188391551499264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196232670632960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2792494377472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2792494377472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196232670632960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48322754 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15937046 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196231996268032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62983863 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44125050 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44125494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111286578 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111306617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188391534459904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3349 _runtime‚2567.748178787 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193440176255488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2791838015488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2931310018560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188391551499264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196232670632960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2792494377472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2792494377472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196232670632960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48322754 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2793313818624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63002343 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196289168734720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63002343 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚193496529294336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28196172 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2932336114688 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2792657443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62982280 _runtime‚2568.422354879 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44137788 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111319048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2782381083817 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191228819674281 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48336770 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193496511291392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48336770 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211384619644928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211384595704832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2783035521777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193496511291392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15942059 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214316955759616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214316774499840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188446455629824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111319048 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63002343 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚193496529294336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28196172 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2932336114688 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2792657443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62982280 _runtime‚2568.422354879 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44137788 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111319048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2782381083817 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191228819674281 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48336770 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193496511291392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48336770 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211384619644928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211384595704832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2783035521777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193496511291392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15942059 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214316955759616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214316774499840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188446455629824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111319048 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2793313818624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63002343 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196289168734720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844à3Ì$2þ- _wandbruntime‚2568  _runtime‚2568Æ¿¦VÿÐ 2loss/SRqqt2/assignment_loss‚1.7223427737668409 2loss/SRqqt4/assignment_loss‚1.5812408366861443 /loss/FRt3/detection_loss‚1.8292250356078148 /loss/FRt4/detection_loss‚2.6510520868523177 -loss/FBt1/detection_loss‚2.88012934923172 /loss/FBt4/detection_loss‚2.7025677462418876 2loss/SRqqt3/assignment_loss‚1.0875690814784982 .loss/FRt1/detection_loss‚3.096958129405975 1loss/SRqqt3/detection_loss‚0.9908851427449421 0loss/SRqqt4/detection_loss‚2.002716574817896 _step‚273 /loss/FRt1/assignment_loss‚3.993037614822388 0loss/FBt3/assignment_loss‚0.2119062135480069 %loss/total_loss‚46.76526203054428 _runtime‚2568.423531078 1loss/FBt1/assignment_loss‚0.17123539429740048 0loss/FBt4/assignment_loss‚0.2978941813732187 epoch‚2 /loss/FRt3/assignment_loss‚1.296959766149521 0loss/FBt2/assignment_loss‚0.7061569811059878 trainer/global_step‚3349 % _timestamp‚1.7884074654361672e+09 0loss/FRt2/assignment_loss‚3.7894539181832916 1loss/SRqqt1/assignment_loss‚2.328487138505335 /loss/FRt2/detection_loss‚2.0813677691751056 .loss/FBt3/detection_loss‚0.550214269598685 0loss/FRt4/assignment_loss‚4.1354957818984985 0loss/SRqqt1/detection_loss‚2.469397105552532 0loss/SRqqt2/detection_loss‚1.947837783370988 /loss/FBt2/detection_loss‚2.2411313561292796è'»ž[€Õ /loss/FRt4/detection_loss‚2.6510520868523177 -loss/FBt1/detection_loss‚2.88012934923172 /loss/FBt4/detection_loss‚2.7025677462418876 2loss/SRqqt3/assignment_loss‚1.0875690814784982 .loss/FRt1/detection_loss‚3.096958129405975 1loss/SRqqt3/detection_loss‚0.9908851427449421 0loss/SRqqt4/detection_loss‚2.002716574817896 _step‚273 /loss/FRt1/assignment_loss‚3.993037614822388 0loss/FBt3/assignment_loss‚0.2119062135480069 %loss/total_loss‚46.76526203054428 _runtime‚2568.423531078 1loss/FBt1/assignment_loss‚0.17123539429740048 0loss/FBt4/assignment_loss‚0.2978941813732187 epoch‚2 /loss/FRt3/assignment_loss‚1.296959766149521 0loss/FBt2/assignment_loss‚0.7061569811059878 trainer/global_step‚3349 % _timestamp‚1.7884074654361672e+09 0loss/FRt2/assignment_loss‚3.7894539181832916 1loss/SRqqt1/assignment_loss‚2.328487138505335 /loss/FRt2/detection_loss‚2.0813677691751056 .loss/FBt3/detection_loss‚0.550214269598685 0loss/FRt4/assignment_loss‚4.1354957818984985 0loss/SRqqt1/detection_loss‚2.469397105552532 0loss/SRqqt2/detection_loss‚1.947837783370988 /loss/FBt2/detection_loss‚2.2411313561292796 2loss/SRqqt2/assignment_loss‚1.7223427737668409 2loss/SRqqt4/assignment_loss‚1.5812408366861443 /loss/FRt3/detection_loss‚1.8292250356078148‘4*©‚(:… µÝãÔö‡‡ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚132.13' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦£ŽR‚‚(:Ç µÝãÔ—ýø  disk.nvme0n1.in‚ 16.6953125 disk.dm-1.in‚0 disk.sda3.out‚ 119.33984375 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2820.703125 disk.rbd1.in‚0 disk.dm-1.out‚ 119.33984375 disk.drbd2265.out‚0 disk.sda.out‚ 119.33984375 disk.sda3.in‚0 disk.sda2.in‚0# disk.nvme0n1.out‚7516.978515625 disk.sda.in‚0 disk.drbd2080.in‚0$ memory_percent‚97.09620475769043 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 16.6953125* proc.memory.percent‚17.216205596923828% disk./.usageGB‚236.24807739257812 disk.rbd1.out‚13.45751953125 disk.drbd2080.out‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚7516.978515625 cpu‚25.284442093578694 network.sent‚ 7.017194e+06 network.recv‚945708* disk./.usagePercent‚15.854806699156002 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2265.in‚0' proc.memory.availableMB‚ 475.7578125 disk.dm-0.out‚7516.978515625 disk.drbd2266.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17´€Î’ƒ‚(:‡ ÄÝãÔòþ’ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.536' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0záƒN„‚(:à ÄÝãÔ¾øŽ  network.recv‚950912% disk./.usageGB‚236.24807739257812 disk.drbd2265.out‚0 disk.sda.out‚ 119.7109375 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-0.out‚7518.134765625 disk.sda2.out‚0 disk.rbd1.out‚13.52001953125 disk.nvme0n1.in‚ 16.6953125# disk.nvme0n1.out‚7518.134765625 network.sent‚ 7.124919e+06 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.dm-1.out‚ 119.7109375$ memory_percent‚97.09477424621582* disk./.usagePercent‚15.854806699156002 disk.drbd2265.in‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 475.9921875! proc.memory.rssMB‚ 2820.703125 cpu‚25.28298107671505 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125* proc.memory.percent‚17.216205596923828 proc.cpu.threads‚17% disk.nvme0c0n1.out‚7518.134765625 disk.sda3.out‚ 119.7109375 disk.dm-1.in‚0/'±2…- _wandbruntime‚2603  _runtime‚2603 Ù›;džÁ & lr-AdamW/pg1‚0.0002484702772999096 & lr-AdamW/pg2‚0.0002484702772999096 trainer/global_step‚3399 % _timestamp‚1.7884075008831475e+09 _runtime‚2603.870603575 _step‚274 ñ²Ì‡Æ _runtime‚2603.870603575 _step‚274 & lr-AdamW/pg1‚0.0002484702772999096 & lr-AdamW/pg2‚0.0002484702772999096 trainer/global_step‚3399 % _timestamp‚1.7884075008831475e+09’ŒÛkÏ2ˆ- _wandbruntime‚2603  _runtime‚2603™—aÝ'‰×O YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196257810194944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63889878 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44762251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112933469 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2822399988261 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2832823821312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199091308381184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63909915 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44761801 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199091308381184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16165397 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193959812453277 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2823054426013 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217380037097984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49023554 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112913430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63889878 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2832823821312 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884075008838868e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196257828197888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49023552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28596406 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49023552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191136758027264 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63909915 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2833480183296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196257828197888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49023554 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214405505790464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199090634016256 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2833480183296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16165845 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚275 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199090634016256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196257810194944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217379849533440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974367683072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1]T†½â'ŠÜO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28596404 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193959140976165 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196257810194944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63889878 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44762251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112933469 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2822399988261 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2832823821312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199091308381184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63909915 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44761801 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199091308381184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16165397 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193959812453277 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2823054426013 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217380037097984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49023554 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112913430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63889878 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2832823821312 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884075008838868e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196257828197888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49023552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28596406 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49023552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191136758027264 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63909915 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2833480183296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196257828197888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49023554 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214405505790464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199090634016256 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2833480183296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16165845 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚275 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199090634016256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196257810194944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217379849533440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974367683072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196314163233792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16169914 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191191645118464 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199148480861184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63928395 _step‚276 PBDeviceStatsMonitor.oG½ë n_train_batch_end/active_bytes.small_pool.peak‚ 863815168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44774918 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49037570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884075015630705e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112965965 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28604572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44774486 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194014861541689 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63928395 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199147806482944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975396348416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191191662157824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63908332 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217442373713920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2823870861185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196314181236736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63908332 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2834299624448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217442552876544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196314163233792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16169914 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191191645118464 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199148480861184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63928395 _step‚276 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44774918 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49037570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884075015630705e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112965965 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28604572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44774486 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194014861541689 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63928395 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199147806482944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975396348416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191191662157824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63908332 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217442373713920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2823870861185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196314181236736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63908332 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2834299624448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217442552876544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3017473319936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193881964555264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚279 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114540282 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64835967 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199075480140288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199075462137344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2863889765421 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64815930 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45397612 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201949271764480 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16392372 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2863235327669 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201949271764480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49724352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29004806 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3449 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201949946129408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2873809627136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64835967 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64815930 GADeviceStatsMonitor.on_train_batch_start/ov¦¦é‡¤ersize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49724354 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2873809627136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884075371715546e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3017320181248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2874465989120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196745182843573 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45397176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49724352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199075480140288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220505515993600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114540282 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64835967 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199075480140288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199075462137344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2863889765421 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64815930 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45397612 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201949271764480 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16392372 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2863235327669 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201949271764480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49724352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29004806 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3449 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201949946129408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2873809627136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64835967 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64815930 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49724354 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2873809627136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884075371715546e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3017320181248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2874465989120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196745182843573 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45397176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49724352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199075480140288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220505515993600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3017473319936—¢CJÈ2- _wandbruntime‚2640  _runtime‚2640 Ä½ºä&žÞM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202007118609408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2864051762633 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217549693411328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3018175495168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64834384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 2640.83386112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16396899 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114572752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202007118609408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193936868685824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29012972 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202006444231168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45409871 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64854447 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114592817 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2875285430272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217549669471232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 # _timestamp‚1.78840753784589e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16397330 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2864706200593 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199131833179136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29012974 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64834384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2874629054976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196800903409097 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193936851646464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199131833179136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49738370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64854447 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3018339106304 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49738370 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29012974 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64834384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2874629054976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196800903409097 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193936851646464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199131833179136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49738370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64854447 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3018339106304 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49738370 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3018175495168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64834384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 2640.83386112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16396899 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114572752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202007118609408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193936868685824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29012972 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202006444231168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45409871 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.·áSš¨large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64854447 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114592817 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2875285430272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217549669471232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 # _timestamp‚1.78840753784589e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16397330 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2864706200593 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944˜æÕ·Ä2 - _wandbruntime‚2640  _runtime‚2640§ÌlV¡Ð /loss/FBt3/detection_loss‚0.6113424881326931 .loss/SRqqt1/assignment_loss‚2.101914280653 2loss/SRqqt4/assignment_loss‚1.6991265736244343 1loss/SRqqt1/detection_loss‚2.5480436126391095 1loss/FBt3/assignment_loss‚0.21170922140336107 1loss/SRqqt2/detection_loss‚1.8304976724916033 .loss/FBt2/detection_loss‚2.619816636045774 trainer/global_step‚3449 % _timestamp‚1.7884075378474653e+09 /loss/FRt2/assignment_loss‚3.594021760899088 2loss/SRqqt3/assignment_loss‚0.8361337945066594 1loss/FBt4/assignment_loss‚0.18877910539352646 .loss/FRt1/detection_loss‚2.992659135111447 2loss/SRqqt2/assignment_loss‚1.6541814944810338 .loss/FRt3/detection_loss‚1.775039941008235 1loss/SRqqt3/detection_loss‚1.0116518996152524 epoch‚2 /loss/FRt2/detection_loss‚2.4397208586983057 /loss/FRt4/detection_loss‚2.3223019361495973 /loss/FBt1/detection_loss‚2.4932775241988048 _runtime‚2640.834838353 _step‚281 .loss/FRt4/assignment_loss‚3.14703644156456 /loss/FBt1/assignment_loss‚0.320876153435425 .loss/FBt4/detection_loss‚3.402086873849233 0loss/FRt3/assignment_loss‚1.9067020436813091 &loss/total_loss‚45.832347191502876 0loss/FRt1/assignment_loss‚3.7894135096977495 0loss/FBt2/assignment_loss‚0.6121499021052538 1loss/SRqqt4/detection_loss‚1.7238643321174163'à[¢Õ 2loss/SRqqt3/assignment_loss‚0.8361337945066594 1loss/FBt4/assignment_loss‚0.18877910539352646 .loss/FRt1/detection_loss‚2.992659135111447 2loss/SRqqt2/assignment_loss‚1.6541814944810338 .loss/FRt3/detection_loss‚1.775039941008235 1loss/SRqqt3/detection_loss‚1.0116518996152524 epoch‚2 /loss/FRt2/detection_loss‚2.4397208586983057 /loss/FRt4/detection_loss‚2.3223019361495973 /loss/FBt1/detection_loss‚2.4932775241988048 _runtime‚2640.834838353 _step‚281 .loss/FRt4/assignment_loss‚3.14703644156456 /loss/FBt1/assignment_loss‚0.320876153435425 .loss/FBt4/detection_loss‚3.402086873849233 0loss/FRt3/assignment_loss‚1.9067020436813091 &loss/total_loss‚45.832347191502876 0loss/FRt1/assignment_loss‚3.7894135096977495 0loss/FBt2/assignment_loss‚0.6121499021052538 1loss/SRqqt4/detection_loss‚1.7238643321174163 /loss/FBt3/detection_loss‚0.6113424881326931 .loss/SRqqt1/assignment_loss‚2.101914280653 2loss/SRqqt4/assignment_loss‚1.6991265736244343 1loss/SRqqt1/detection_loss‚2.5480436126391095 1loss/FBt3/assignment_loss‚0.21170922140336107 1loss/SRqqt2/detection_loss‚1.8304976724916033 .loss/FBt2/detection_loss‚2.619816636045774 trainer/global_step‚3449 % _timestamp‚1.7884075378474653e+09 /loss/FRt2/assignment_loss‚3.594021760899088™¯¢´M£‚(:’ €ÞãÔ¬ã gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.007' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.401400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x¥×XL¤‚(:Á €ÞãÔÙ†ú  disk.sda.out‚ 121.734375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0" disk.dm-0.out‚7520.99658203125 disk.sda.in‚0 disk.drbd2266.in‚0$ memory_percent‚97.09663391113281 disk.sda1.out‚0 disk.rbd1.in‚0 network.recv‚969003 disk.rbd1.out‚ 13.802734375 disk.drbd2265.out‚0 disk.sda3.out‚ 121.734375$ proc.memory.availableMB‚475.6875% disk./.usageGB‚236.24737930297852 disk.nvme0c0n1.in‚ 16.6953125' disk.nvme0c0n1.out‚7520.99658203125 disk.nvme0n1.in‚ 16.6953125 disk.drbd2080.in‚0 disk.dm-0.in‚ 16.6953125 disk.dm-1.in‚0 disk.dm-1.out‚ 121.734375 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2820.703125* proc.memory.percent‚17.216205596923828 network.sent‚ 7.249816e+06 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚0 cpu‚25.27769463022766* disk./.usagePercent‚15.854759849777237% disk.nvme0n1.out‚7520.99658203125áë6‘¥‚(:† ÞãÔü¤ gpu.0.gpu‚4.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&D¤*Q¦‚(:Æ ÞãÔŸ¢ü  disk.rbd1.out‚ 13.888671875 disk.nvme0n1.in‚ 16.6953125% disk./.usageGB‚236.24738311767578 disk.sda3.out‚ 122.2890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 network.sent‚ 7.358253e+06 network.recv‚974978 disk.drbd1716.out‚0" disk.dm-0.out‚7521.22314453125 disk.dm-1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 475.703125* proc.memory.percent‚17.216205596923828 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 16.6953125' disk.nvme0c0n1.out‚7521.22314453125 disk.dm-1.out‚ 122.2890625) disk./.usagePercent‚15.85476010578477 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 16.6953125 cpu‚25.276334180245478 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0$ memory_percent‚97.09653854370117 disk.drbd1716.in‚0% disk.nvme0n1.out‚7521.22314453125 disk.sda.in‚0 disk.sda.out‚ 122.2890625 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2820.703125 proc.cpu.threads‚17 disk.sda2.out‚0Ýý®ã2§- _wandbruntime‚2676  _runtime‚2676V‘ýÜĨ¾ _runtime‚2676.315728798 _step‚282 % lr-AdamW/pg1‚0.000246077612712871 % lr-AdamW/pg2‚0.000246077612712871 trainer/global_step‚3499 $ _timestamp‚1.788407573328296e+09`ºüçÉ©Ã % lr-AdamW/pg2‚0.000246077612712871 trainer/global_step‚3499 $ _timestamp‚1.788407573328296e+09 _runtime‚2676.315728798 _step‚282 % lr-AdamW/pg1‚0.000246077612712871š_ â2ª- _wandbruntime‚2676  _runtime‚2676-h¯Ý'«×O ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223631233164288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223631045599744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220570579556864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2915451794944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46033840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16620634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50425152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116167134 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2915451794944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50425154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65762019 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204808583877632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199531896188093 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199531224710981 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65741982 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204807909512704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201893132082688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884075733289864e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116187173 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201893114079744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29413204 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196627171083264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204808583877632 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚3499 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2914795432960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2—xâ¨â'¬ÜO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116167134 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2915451794944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50425154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65762019 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204808583877632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199531896188093 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199531224710981 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65741982 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204807909512704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201893132082688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884075733289864e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116187173 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201893114079744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29413204 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196627171083264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204808583877632 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚3499 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2914795432960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223631233164288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223631045599744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220570579556864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2915451794944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46033840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16620634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50425152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0›tÃè•2­- _wandbruntime‚2677  _runtime‚2677¸F“X®àM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚204865756357632 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚201949485121536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2904887102041 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚201949467118592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199586945276505 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196682075213824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50439170 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65780499 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50439168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116199604 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2915614860800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788407574016754e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2916271236096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3061353484800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65760436 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199587616753825 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚201949485121536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2916271236096 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚284 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚“nq¿Ž65780499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2905541540001 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201949467118592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50439168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚2677.004816676 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196682058174464 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚3499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46046029 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29421372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223693747390464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220632230294528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2915614860800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116219669 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50439170 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16625104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199587616753825 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚201949485121536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2916271236096 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚284 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65780499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2905541540001 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201949467118592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50439168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚2677.004816676 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196682058174464 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚3499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46046029 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29421372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223693747390464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220632230294528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2915614860800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116219669 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50439170 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16625104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚201949467118592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199586945276505 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196682075213824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50439170 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65780499 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50439168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116199604 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2915614860800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788407574016754e+09œ‹ës2°- _wandbruntime‚2677  _runtime‚2677ôs^±Ø /loss/FRt4/assignment_loss‚3.032790527989467 1loss/SRqqt3/assignment_loss‚1.197783883613876 /loss/FRt1/detection_loss‚3.2492056946669305 /loss/FRt2/detection_loss‚2.1620208449088656 trainer/global_step‚3499 2loss/SRqqt1/assignment_loss‚2.3189891382403998 1loss/FBt4/assignment_loss‚0.05497559959985665 1loss/SRqqt1/detection_loss‚3.0685570538043976 _runtime‚2677.005869192 2loss/SRqqt2/assignment_loss‚2.0520845072136984 2loss/SRqqt4/assignment_loss‚1.4693201171748083 .loss/FBt2/detection_loss‚2.510615346687181 /loss/FBt3/detection_loss‚0.8806691944482954 %loss/total_loss‚49.09927851131346 0loss/FRt3/assignment_loss‚2.0238727735621587 1loss/FBt3/assignment_loss‚0.16649158180850557 /loss/FRt3/detection_loss‚1.8019681846989053 1loss/SRqqt3/detection_loss‚1.0087684765458107 0loss/FRt2/assignment_loss‚3.5153995488698664 0loss/SRqqt2/detection_loss‚1.903299574719535 epoch‚2 _step‚285 1loss/SRqqt4/detection_loss‚1.5279258715695347 .loss/FBt4/detection_loss‚5.021147549152374 0loss/FRt1/assignment_loss‚3.9412436740739003 1loss/FBt1/assignment_loss‚0.16646714843227528 1loss/FBt2/assignment_loss‚0.36866028816023444 /loss/FRt4/detection_loss‚2.2171386098489165 /loss/FBt1/detection_loss‚3.4398833215236664 % _timestamp‚1.7884075740184958e+096FÛc²Ý /loss/FRt4/assignment_loss‚3.032790527989467 1loss/SRqqt3/assignment_loss‚1.197783883613876 /loss/FRt1/detection_loss‚3.2492056946669305 /loss/FRt2/detection_loss‚2.1620208449088656 trainer/global_step‚3499 2loss/SRqqt1/assignment_loss‚2.3189891382403998 1loss/FBt4/assignment_loss‚0.05497559959985665 1loss/SRqqt1/detection_loss‚3.0685570538043976 _runtime‚2677.005869192 2loss/SRqqt2/assignment_loss‚2.0520845072136984 2loss/SRqqt4/assignment_loss‚1.4693201171748083 .loss/FBt2/detection_loss‚2.510615346687181 /loss/FBt3/detection_loss‚0.8806691944482954 %loss/total_loss‚49.09927851131346 0loss/FRt3/assignment_loss‚2.0238727735621587 1loss/FBt3/assignment_loss‚0.16649158180850557 /loss/FRt3/detection_loss‚1.8019681846989053 1loss/SRqqt3/detection_loss‚1.0087684765458107 0loss/FRt2/assignment_loss‚3.5153995488698664 0loss/SRqqt2/detection_loss‚1.903299574719535 epoch‚2 _step‚285 1loss/SRqqt4/detection_loss‚1.5279258715695347 .loss/FBt4/detection_loss‚5.021147549152374 0loss/FRt1/assignment_loss‚3.9412436740739003 1loss/FBt1/assignment_loss‚0.16646714843227528 1loss/FBt2/assignment_loss‚0.36866028816023444 /loss/FRt4/detection_loss‚2.2171386098489165 /loss/FBt1/detection_loss‚3.4398833215236664 % _timestamp‚1.7884075740184958e+09’ËÌ9’³‚(:‡ žÞãÔË·ƒ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚137.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁÿS!P´‚(:Å žÞãÔÌåý  disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125" disk.dm-0.out‚7521.60595703125 disk.drbd2265.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 475.9140625 network.sent‚ 7.364088e+06 network.recv‚979030 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 14.044921875 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚7521.60595703125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 16.6953125$ memory_percent‚97.09525108337402 disk.dm-1.out‚ 122.7890625* disk./.usagePercent‚15.854822571623126$ disk./.usageGB‚236.2483139038086 disk.sda3.out‚ 122.7890625! proc.memory.rssMB‚ 2820.703125 cpu‚25.2750811847756 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 122.7890625 disk.sda3.in‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚7521.60595703125 disk.sda.in‚0* proc.memory.percent‚17.216205596923828 proc.cpu.threads‚17ïÇ–Æœµ‚(:‘ ­ÞãÔÊφ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.63' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.525999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m-}_T¶‚(:É ­ÞãÔƒ‡õ  disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚7521.9091796875! proc.memory.rssMB‚ 2820.703125 cpu‚25.273824405320855* disk./.usagePercent‚15.854820523562852 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚ 16.6953125 disk.drbd1716.out‚0 network.sent‚ 7.472917e+06 disk.sda.out‚ 123.44921875 disk.sda3.out‚ 123.44921875 disk.drbd2266.in‚0 disk.rbd1.out‚ 14.044921875$ memory_percent‚97.09675312042236 disk.dm-1.in‚0 disk.dm-1.out‚ 123.44921875 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚7521.9091796875 disk.rbd1.in‚0* proc.memory.percent‚17.216205596923828 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 16.6953125 proc.cpu.threads‚17 network.recv‚985648% disk./.usageGB‚236.24828338623047 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚ 16.6953125! disk.dm-0.out‚7521.9091796875( proc.memory.availableMB‚ 475.66796875W¢„2·- _wandbruntime‚2712  _runtime‚271238§È¸Â _runtime‚2712.617559011 _step‚286 ' lr-AdamW/pg1‚0.00024484722268254685 ' lr-AdamW/pg2‚0.00024484722268254685 trainer/global_step‚3549 $ _timestamp‚1.788407609630112e+099cÍ¹Ç $ _timestamp‚1.788407609630112e+09 _runtime‚2712.617559011 _step‚286 ' lr-AdamW/pg1‚0.00024484722268254685 ' lr-AdamW/pg2‚0.00024484722268254685 trainer/global_step‚3549žÕVPú2º- _wandbruntime‚2712  _runtime‚2712D— ÖÜ'»ÖO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51125952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2955781238784 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51125952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202317266578389 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204710784025088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51125954 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚287 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2944906006485 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚3549 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199372377611264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117814025 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199372360571904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207667221625856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207667221625856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207666547260928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204710766022144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956437600768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2945560444237 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66688071 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204710784025088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66688071 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66668034 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66668034 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226756863156736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3103746716672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202317938055501 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204710766022144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207666547260928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51125952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202317266578389 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204710784025088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51125954 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚287 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2944906006485 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚3549 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199372377611264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117814025 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199372360571904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207667221625856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207667221625856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207666547260928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204710766022144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956437600768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2945560444237 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66688071 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204710784025088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66688071 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66668034 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66668034 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226756863156736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3103746716672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202317938055501 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204710766022144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207666547260928 JVç&¾áM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207724394105856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66686488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51139968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16852555 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46682790 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51139968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202372987143913 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946376879409 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2956600666624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957257041920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3104597976576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204767119060992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2957257041920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223714767177728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2945722441449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2956600666624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226819162923008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199427281741824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66706551 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66686488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29829772 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3104419685376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204767137063936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46682327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16853016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207723719727616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226819365154304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204767137063936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207724394105856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66706551 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117826456X…ºVì&¿æM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66686488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51139968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16852555 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46682790 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51139968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202372987143913 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946376879409 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2956600666624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957257041920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3104597976576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204767119060992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2957257041920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223714767177728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2945722441449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2956600666624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226819162923008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199427281741824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66706551 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66686488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29829772 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3104419685376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204767137063936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46682327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16853016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207723719727616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226819365154304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204767137063936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207724394105856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66706551 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117826456 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207724394105856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 qËñ2À- _wandbruntime‚2713  _runtime‚2713€Ås%XÁÒ 1loss/SRqqt4/detection_loss‚1.6965577971061756 .loss/FBt1/detection_loss‚3.601661417633295 trainer/global_step‚3549 1loss/SRqqt2/detection_loss‚2.2266841930421917 /loss/FBt2/detection_loss‚2.0474648118019103 %loss/total_loss‚49.06939037411573 2loss/SRqqt2/assignment_loss‚1.5572276926514776 .loss/FRt4/detection_loss‚2.525921379774809 .loss/FBt4/detection_loss‚3.314515312512716 _runtime‚2713.302098836 2loss/SRqqt1/assignment_loss‚1.7968963381762688 0loss/FBt4/assignment_loss‚1.3094968853984028 epoch‚2 % _timestamp‚1.7884076103147385e+09 0loss/FRt4/assignment_loss‚3.5608387321233748 0loss/SRqqt3/detection_loss‚1.060257099096624 /loss/FBt3/detection_loss‚0.5333242085963533 .loss/FRt1/detection_loss‚3.228942167758942 0loss/FRt2/assignment_loss‚3.1439779761225677 2loss/SRqqt3/assignment_loss‚1.2762318539734878 0loss/FBt1/assignment_loss‚0.5647864013121762 .loss/FRt3/detection_loss‚1.599504029280261 /loss/FRt1/assignment_loss‚4.628128423690796 0loss/FRt3/assignment_loss‚1.7281556892813297 2loss/SRqqt4/assignment_loss‚1.5332837945877993 _step‚289 0loss/FBt2/assignment_loss‚0.9891832992434426 1loss/FBt3/assignment_loss‚0.36982524012409024 .loss/FRt2/detection_loss‚2.119562323703322 1loss/SRqqt1/detection_loss‚2.6569633071239176ñõ½]Â× 2loss/SRqqt1/assignment_loss‚1.7968963381762688 0loss/FBt4/assignment_loss‚1.3094968853984028 epoch‚2 % _timestamp‚1.7884076103147385e+09 0loss/FRt4/assignment_loss‚3.5608387321233748 0loss/SRqqt3/detection_loss‚1.060257099096624 /loss/FBt3/detection_loss‚0.5333242085963533 .loss/FRt1/detection_loss‚3.228942167758942 0loss/FRt2/assignment_loss‚3.1439779761225677 2loss/SRqqt3/assignment_loss‚1.2762318539734878 0loss/FBt1/assignment_loss‚0.5647864013121762 .loss/FRt3/detection_loss‚1.599504029280261 /loss/FRt1/assignment_loss‚4.628128423690796 0loss/FRt3/assignment_loss‚1.7281556892813297 2loss/SRqqt4/assignment_loss‚1.5332837945877993 _step‚289 0loss/FBt2/assignment_loss‚0.9891832992434426 1loss/FBt3/assignment_loss‚0.36982524012409024 .loss/FRt2/detection_loss‚2.119562323703322 1loss/SRqqt1/detection_loss‚2.6569633071239176 1loss/SRqqt4/detection_loss‚1.6965577971061756 .loss/FBt1/detection_loss‚3.601661417633295 trainer/global_step‚3549 1loss/SRqqt2/detection_loss‚2.2266841930421917 /loss/FBt2/detection_loss‚2.0474648118019103 %loss/total_loss‚49.06939037411573 2loss/SRqqt2/assignment_loss‚1.5572276926514776 .loss/FRt4/detection_loss‚2.525921379774809 .loss/FBt4/detection_loss‚3.314515312512716 _runtime‚2713.302098836¡ç;,ёÂ(:† ¼ÞãÔð“ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.065' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.413 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ¸³«çÄ‚(:Ä ¼ÞãÔñÒ  disk.drbd2080.out‚0 disk.sda1.out‚0 cpu‚25.272522875484405 proc.cpu.threads‚17 network.sent‚ 7.478158e+06 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 123.8828125È«µAh disk.drbd2265.out‚0$ memory_percent‚97.09835052490234* proc.memory.percent‚17.216205596923828$ disk.nvme0n1.out‚7522.7626953125* disk./.usagePercent‚15.854822315615591! disk.dm-0.out‚7522.7626953125 disk.nvme0c0n1.in‚ 16.6953125& disk.nvme0c0n1.out‚7522.7626953125 disk.dm-0.in‚ 16.6953125 disk.sda.out‚ 123.8828125 disk.sda3.out‚ 123.8828125 disk.rbd0.in‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 475.40625% disk./.usageGB‚236.24831008911133 disk.rbd1.in‚0 disk.rbd1.out‚ 14.0771484375 disk.nvme0n1.in‚ 16.6953125 disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.dm-1.in‚0! proc.memory.rssMB‚ 2820.703125 network.recv‚989243 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0%¥P’Å‚(:‡ ËÞãÔ´ö gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tà ?Æ‚(:´ ËÞãÔÏúý ! disk.dm-0.out‚7523.0712890625 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚7523.0712890625 disk.drbd2265.in‚0 disk.rbd0.out‚0 network.sent‚ 7.586724e+06 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 16.6953125 disk.rbd0.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 475.3515625! proc.memory.rssMB‚ 2820.703125 disk.sda.out‚124.5 disk.sda3.out‚124.5 disk.drbd2266.out‚0% disk./.usageGB‚236.24832916259766 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0* proc.memory.percent‚17.216205596923828 disk.rbd1.out‚ 14.0771484375* disk./.usagePercent‚15.854823595653261 disk.nvme0c0n1.in‚ 16.6953125 disk.dm-1.in‚0 disk.drbd2265.out‚0$ memory_percent‚97.09868431091309 cpu‚25.271204254479326$ disk.nvme0n1.out‚7523.0712890625 disk.dm-1.out‚124.5 disk.drbd2266.in‚0 network.recv‚995655 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 16.6953125¼Óý’Ç‚(:‡ ÚÞãÔµû˜ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^`êéEÈ‚(:º ÚÞãÔ¦¾  disk.drbd1716.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0! disk.nvme0n1.out‚ 7523.5546875 network.sent‚ 7.59251e+06 disk.nvme0c0n1.in‚ 16.6953125 disk.rbd1.in‚0 disk.dm-1.out‚ 125.0546875 disk.drbd2080.in‚0* disk./.usagePercent‚15.854833835954633 disk.dm-0.in‚ 16.6953125 disk.dm-0.out‚ 7523.5546875 network.recv‚999805 disk.sda3.out‚ 125.0546875 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 14.2333984375 disk.sda.in‚0* proc.memory.percent‚17.216205596923828% disk./.usageGB‚236.24848175048828# disk.nvme0c0n1.out‚ 7523.5546875 disk.drbd1716.in‚0 disk.sda3.in‚0 disk.drbd2266.out‚0$ memory_percent‚97.09720611572266% proc.memory.availableMB‚ 475.59375 disk.sda.out‚ 125.0546875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 cpu‚25.269944536052993 proc.cpu.threads‚17 disk.nvme0n1.in‚ 16.6953125 disk.drbd2265.in‚0 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2820.703125€Š2É- _wandbruntime‚2748  _runtime‚2748øbÇÈÊ _runtime‚2748.479567344 _step‚290 ' lr-AdamW/pg1‚0.00024359446780485816 ' lr-AdamW/pg2‚0.00024359446780485816 trainer/global_step‚3599 $ _timestamp‚1.788407645492142e+09 ²¹ÉÍËÇ $ _timestamp‚1.788407645492142e+09 _runtime‚2748.479567344 _step‚290 ' lr-AdamW/pg1‚0.00024359446780485816 ' lr-AdamW/pg2‚0.00024359446780485816 trainer/global_step‚3599¢Œ›+U2Ì- _wandbruntime‚2748  _runtime‚2748³M&Ý'Í×O JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚2748.480473323 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47306521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2997423406592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2996767044608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119420838 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51826754 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2986395783645 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67594086 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229882210554880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207528435967488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚229882402313728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚3599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2997423406592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚210525185009152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210525185009152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17076068 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202117567099904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30230006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207528417964544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚205103979922909 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 % _timestamp‚1.7884076454928288e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67614123 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67594086 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17076515 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202117584139264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1Ï!Ÿâ'ÎÜO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226735629383168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67614123 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚210525859374080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51826754 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚291 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51826752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226735653323264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3146748990464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205103308445797 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2985741345893 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202117567099904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30230006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207528417964544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚205103979922909 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 % _timestamp‚1.7884076454928288e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67614123 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67594086 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17076515 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202117584139264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚2748.480473323 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47306521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2997423406592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2996767044608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119420838 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51826754 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2986395783645 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67594086 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229882210554880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207528435967488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚229882402313728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚3599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2997423406592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚210525185009152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210525185009152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17076068 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚210583031854080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207584789006336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚2749.163538333 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987212218817 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207584771003392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67632603 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229944728894464 % _timestamp‚1.7884076461755643e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207584789006336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2997586472448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17080569 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202172488269824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚210583031854080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47318741 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17081006 _step‚292 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67612540 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51840770 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51840770 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119473373 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226797280120832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3147608190464 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚£•ŸÄ<0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67612540 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202172471230464 trainer/global_step‚3599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2998242847744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚210582357475840 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844Š­Üüì&ÑæM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207584789006336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2997586472448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17080569 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202172488269824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚210583031854080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47318741 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17081006 _step‚292 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67612540 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51840770 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51840770 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119473373 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226797280120832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3147608190464 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67612540 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202172471230464 trainer/global_step‚3599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2998242847744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚210582357475840 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2986557780857 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚210583031854080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207584789006336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚2749.163538333 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987212218817 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207584771003392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67632603 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229944728894464 % _timestamp‚1.7884076461755643e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240¤×Ï!ë2Ò- _wandbruntime‚2749  _runtime‚2749Ÿ¶—LUÓÏ /loss/FBt3/detection_loss‚0.8266319963758055 2loss/SRqqt3/assignment_loss‚1.4778267569839953 0loss/FBt1/assignment_loss‚0.4771913497285709 1loss/SRqqt2/detection_loss‚1.7188478177785873 .loss/FRt1/assignment_loss‚3.53429122765859 .loss/FRt1/detection_loss‚2.794377088546753 %loss/total_loss‚45.82874136876791 % _timestamp‚1.7884076461774201e+09 0loss/FRt4/assignment_loss‚3.2885127078403125 1loss/SRqqt2/assignment_loss‚1.648038625717163 1loss/SRqqt3/detection_loss‚1.0280949668160506 1loss/SRqqt4/detection_loss‚1.6831503726646935 _step‚293 /loss/FRt3/assignment_loss‚1.641147254601769 1loss/FBt4/assignment_loss‚0.15909120440483093 /loss/FBt1/detection_loss‚2.8673816435039043 /loss/FBt2/detection_loss‚2.2336274568851175 /loss/FRt2/assignment_loss‚3.649934550890556 2loss/SRqqt1/assignment_loss‚2.2726344101958804 0loss/FBt3/assignment_loss‚0.3943506265292497 -loss/FRt2/detection_loss‚2.22302794112609 _runtime‚2749.164795585 /loss/FRt3/detection_loss‚1.8597178781486077 0loss/SRqqt1/detection_loss‚2.245683032605383 epoch‚2 0loss/FBt2/assignment_loss‚0.4672096148133147 /loss/FBt4/detection_loss‚3.7900073726971946 trainer/global_step‚3599 1loss/SRqqt4/assignment_loss‚1.346451849689786 .loss/FRt4/detection_loss‚2.201513622565703ÈùòZÔÔ 1loss/SRqqt4/assignment_loss‚1.346451849689786 .loss/FRt4/detection_loss‚2.201513622565703 /loss/FBt3/detection_loss‚0.8266319963758055 2loss/SRqqt3/assignment_loss‚1.4778267569839953 0loss/FBt1/assignment_loss‚0.4771913497285709 1loss/SRqqt2/detection_loss‚1.7188478177785873 .loss/FRt1/assignment_loss‚3.53429122765859 .loss/FRt1/detection_loss‚2.794377088546753 %loss/total_loss‚45.82874136876791 % _timestamp‚1.7884076461774201e+09 0loss/FRt4/assignment_loss‚3.2885127078403125 1loss/SRqqt2/assignment_loss‚1.648038625717163 1loss/SRqqt3/detection_loss‚1.0280949668160506 1loss/SRqqt4/detection_loss‚1.6831503726646935 _step‚293 /loss/FRt3/assignment_loss‚1.641147254601769 1loss/FBt4/assignment_loss‚0.15909120440483093 /loss/FBt1/detection_loss‚2.8673816435039043 /loss/FBt2/detection_loss‚2.2336274568851175 /loss/FRt2/assignment_loss‚3.649934550890556 2loss/SRqqt1/assignment_loss‚2.2726344101958804 0loss/FBt3/assignment_loss‚0.3943506265292497 -loss/FRt2/detection_loss‚2.22302794112609 _runtime‚2749.164795585 /loss/FRt3/detection_loss‚1.8597178781486077 0loss/SRqqt1/detection_loss‚2.245683032605383 epoch‚2 0loss/FBt2/assignment_loss‚0.4672096148133147 /loss/FBt4/detection_loss‚3.7900073726971946 trainer/global_step‚3599¥5?…’Õ‚(:‡ éÞãÔóœ¢ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ììÕåEÖ‚(:º éÞãÔ¸âÄ  disk.rbd1.in‚0 disk.sda.out‚125.8125 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 14.2333984375! proc.memory.rssMB‚ 2820.703125 disk.nvme0n1.in‚ 16.6953125 network.sent‚ 7.598797e+06* disk./.usagePercent‚15.854844332263534% disk./.usageGB‚236.24863815307617 disk.sda3.in‚0 disk.sda3.out‚125.8125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚125.8125" disk.nvme0n1.out‚ 7523.85546875* proc.memory.percent‚17.216205596923828 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚ 16.6953125$ disk.nvme0c0n1.out‚ 7523.85546875 disk.dm-0.out‚ 7523.85546875 cpu‚25.268721172668865 network.recv‚ 1.004153e+06 disk.sda.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0$ proc.memory.availableMB‚475.3125 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 16.6953125$ memory_percent‚97.09892272949219 proc.cpu.threads‚17;<~ƒ×‚(:’ øÞãÔõÌ gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.474' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.294799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"VØ‚(:Ë øÞãÔÐÁÚ  disk.sda1.out‚0% disk.nvme0n1.out‚7524.26708984375" disk.dm-0.out‚7524.26708984375* disk./.usagePercent‚15.854843052225863 disk.dm-1.out‚ 126.359375 disk.drbd2265.out‚0 disk.rbd1.out‚ 14.3818359375 disk.nvme0n1.in‚ 16.6953125 proc.cpu.threads‚17 network.recv‚ 1.009723e+06 disk.sda.in‚0 disk.sda3.out‚ 126.359375 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2820.703125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 16.6953125' disk.nvme0c0n1.out‚7524.26708984375* proc.memory.percent‚17.216205596923828 network.sent‚ 7.706483e+06 disk.dm-1.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 475.5546875% disk./.usageGB‚236.24861907958984 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.09744453430176 cpu‚25.27794082622402 disk.sda1.in‚0 disk.sda.out‚ 126.359375 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0½+T{œÙ‚(:‘ ‡ßãÔ¶¯ï gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.341' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.868199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚Ш³QÚ‚(:Æ ‡ßãÔŠÂÔ  disk.rbd0.in‚0 network.sent‚ 7.712559e+06 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 127.0234375 disk.drbd2080.in‚0 disk.sda2.out‚0% disk./.usageGB‚236.24863815307617 disk.rbd1.in‚0 disk.nvme0n1.in‚ 16.6953125 disk.nvme0c0n1.in‚ 16.6953125* disk./.usagePercent‚15.854844332263534 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 127.0234375! proc.memory.rssMB‚ 2820.703125# disk.nvme0n1.out‚7524.884765625 disk.dm-0.in‚ 16.6953125 disk.dm-1.out‚ 127.0234375 disk.sda.in‚0 disk.sda3.in‚0% disk.nvme0c0n1.out‚7524.884765625$ memory_percent‚97.09739685058594 network.recv‚ 1.013762e+06 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0$ proc.memory.availableMB‚475.5625* proc.memory.percent‚17.216205596923828 cpu‚25.303528901213888 disk.rbd1.out‚ 14.3818359375 disk.dm-0.out‚7524.884765625:á=Ì‘Û‚(:† –ßãÔƹú gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚118.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”w3¢OÜ‚(:Ä –ßãÔëøå  disk.drbd2080.out‚0* disk./.usagePercent‚15.854844332263534 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0% disk./.usageGB‚236.24863815307617 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 16.6953125 disk.dm-1.out‚ 127.44921875$ memory_percent‚97.09296226501465 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd1716.out‚0' proc.memory.availableMB‚ 476.2890625 disk.drbd1716.in‚0 disk.rbd1.out‚ 14.6240234375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.out‚ 7525.8671875 network.sent‚ 7.718508e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 7525.8671875 proc.memory.rssMB‚ 2819.71875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 16.6953125 cpu‚25.329390716197395 disk.sda.in‚0 disk.sda.out‚ 127.44921875 disk.dm-0.out‚ 7525.8671875) proc.memory.percent‚17.21019744873047 network.recv‚ 1.018087e+06 disk.sda3.in‚0 disk.sda3.out‚ 127.44921875Ÿ"›2Ý- _wandbruntime‚2816  _runtime‚2816Ø-D*Þ¾T PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚210910992466432 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚178 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚210910974463488 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 121237802 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121221824 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585536 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚68473130 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚233506747686912 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚205384276970496 _step‚294 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚213948760022528 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815168 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚52748696 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 3026602812437 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3037785559040 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 3038278388736 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚48197520 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699021 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 3027093726825 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510832640 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚30777404 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚208411387736681 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚208410879782933 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15978 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953748 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815168 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚210910974463488 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚213949270855168 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829696 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121221824 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17419938 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚230317784919552 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134218752 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚176 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚233506589528064 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚3626 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚52748694 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚68489106 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚48197342 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15976 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚210910992466432 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510832640 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15976 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3188828548608 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚52748694 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914388 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚68473130 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24127 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 158158848 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24400 % _timestamp‚1.7884077136340618e+09 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚52748696 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3188962767360 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚213949270855168 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492829696 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853517 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3037785559040 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚205384294009856 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 121237802 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚30777406 _runtime‚2816.621923031 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510832640 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚30777404 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚208411387736681 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚208410879782933 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15978 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953748 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815168 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚210910974463488 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚213949270855168 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829696 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121221824 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17419938 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚230317784919552 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134218752 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚176 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚233506589528064 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚3626 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚52748694 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚68489106 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚48197342 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15976 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚210910992466432 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510832640 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15976 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3188828548608 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚52748694 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914388 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚68473130 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24127 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 158158848 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24400 % _timestamp‚1.7884077136340618e+09 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚52748696 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3188962767360 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚213949270855168 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492829696 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853517 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3037785559040 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚205384294009856 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 121237802 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚30777406 _runtime‚2816.621923031 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚210910974463488 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 121237802 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121221824 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585536 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚68473130 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚233506747686912 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚205384276970496 _step‚294 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚213948760022528 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815168 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚52748696 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 3026602812437 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872¦ ¿ƒ…2à- _wandbruntime‚2817  _runtime‚2817ÈÓ tN)áÈR SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚52759502 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 3038643515392 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚210947804492288 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚68480432 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚17424248 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3189327894016 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚48208335 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚30783908 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚208446537151829 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚213986466010624 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829696 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚233544988547072 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚233544830388224 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 121255910 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3038643515392 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚230355660653056 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15978 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚52759502 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚52759500 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829696 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17424425 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚213985955177984 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚177 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699021 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚52759500 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚210947822495232 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚208447045105577 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 121255910 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚210947822495232 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3027457336233 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚68480432 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚213985955177984 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24400 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832640 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832640 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15978 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815168 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24127 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 _runtime‚2817.534848222 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚213986466010624 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚48208156 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚30783910 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585536 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 3038150685696 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3026966421845Ø ’S)âÍR TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚48208335 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚30783908 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚208446537151829 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚213986466010624 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829696 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚233544988547072 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚233544830388224 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 121255910 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3038643515392 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚230355660653056 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15978 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚52759502 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚52759500 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829696 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17424425 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚213985955177984 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚177 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699021 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚52759500 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚210947822495232 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚208447045105577 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 121255910 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚210947822495232 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3027457336233 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚68480432 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚213985955177984 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24400 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832640 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832640 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15978 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815168 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24127 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 _runtime‚2817.534848222 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚213986466010624 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚48208156 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚30783910 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585536 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 3038150685696 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3026966421845 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚52759502 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 3038643515392 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚210947804492288 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚68480432 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚17424248 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3189327894016§£5GRã‚(:… ¥ßãÔœ·Ž gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚186.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á·‚Jä‚(:¿ ¥ßãÔ¬ïç  disk.sda.in‚0 disk.rbd0.in‚0# memory_percent‚97.0916748046875! proc.memory.availableMB‚476.5 cpu‚25.35278533135038 disk.dm-0.in‚ 16.6953125 disk.drbd2266.out‚0$ disk.nvme0c0n1.out‚ 7526.08984375 disk.sda3.out‚ 128.28515625 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2819.71875 disk.nvme0c0n1.in‚ 16.6953125* disk./.usagePercent‚15.854843052225863 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 16.6953125 disk.sda.out‚ 128.28515625 disk.sda1.in‚0% disk./.usageGB‚236.24861907958984" disk.nvme0n1.out‚ 7526.08984375 disk.dm-0.out‚ 7526.08984375 disk.dm-1.out‚ 128.28515625 disk.sda3.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.21019744873047 network.sent‚ 7.724782e+06 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 14.6240234375 network.recv‚ 1.022466e+06Xvúó2å- _wandbruntime‚2830  _runtime‚2830Á+Ê<æè§ ("Purity/3FRt1SRq]ò.%ùqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.289814829826355 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24112597107887268 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0672827661037445 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14174750447273254 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_6_of_6‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.05319149047136307 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 jet/accuracy_1_of_7‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996938169002533 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13996562361717224 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9887680411338806 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4818452298641205 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.495807021856308 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011659108102321625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08543571829795837 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12163742631673813 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36034226417541504 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.35327738523483276 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4973912537097931 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4388306438922882 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018674001097679138 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5216559171676636 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3777778148651123 *jet/accuracy_1_of_5‚0.9509536027908325 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9914838075637817 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.40689656138420105 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43845319747924805 particle/accuracy_5_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5179547667503357 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.44595959782600403 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4861111640930176 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4980917274951935 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4863683879375458 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022192273288965225 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_1_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.39506176114082336 particle/accuracy_1_of_6‚1 jet/accuracy_2_of_7‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9954047799110413 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25965818762779236 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4674477279186249 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2622280716896057 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4886071979999542 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46795985102653503 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23238733410835266 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_1_of_6‚0.9615384340286255 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.2966934144496918 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22420856356620789 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12969201803207397 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_7‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.500788688659668 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.48020488023757935 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 )loss/val_total_loss‚46.93182296504428 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4051200747489929 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9847952723503113 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3728395104408264 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022249136120080948 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04518524557352066 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5033667087554932 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49968722462654114 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1275961846113205 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.48936927318573 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5079364776611328 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024548934772610664 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3219163417816162 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9894781112670898 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15567342936992645 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_2_of_5‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39444440603256226 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333335638046265 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5063778758049011 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23339548707008362 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9843990206718445 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2083333432674408 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4600718319416046 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011749234981834888 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4276272654533386 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.49965283274650574 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1350667029619217 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_4_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32545432448387146 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25057467818260193 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4540540874004364 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _step‚296 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8064900636672974 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.531549870967865 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002858693478628993 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 jet/accuracy_5_of_6‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5083353519439697 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48306381702423096 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1149425357580185 jet/accuracy_3_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3701550364494324 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6364431381225586 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23695434629917145 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4696327745914459 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.061403512954711914 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17787349224090576 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1707923412322998 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5116744041442871 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3616279363632202 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.49407628178596497 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08137982338666916 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5030678510665894 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9907373189926147 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993182897567749 particle/accuracy_3_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 trainer/global_step‚3626 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9885729551315308 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20675250887870789 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4901207387447357 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.2448783963918686 (particle/f_score‚0.40554460883140564 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 _runtime‚ 2830.52152664 *jet/accuracy_4_of_5‚0.0098566310480237 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 *jet/accuracy_2_of_6‚0.4038461446762085 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4597175717353821 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5648148059844971 particle/accuracy_6_of_7‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870175123214722 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20049557089805603 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905905723571777 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14223773777484894 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13843098282814026 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4030245244503021 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4246266186237335 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06321839243173599 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4437602460384369 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1377105414867401 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 epoch‚2 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4840024709701538 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9509766697883606 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 jet/accuracy_6_of_6‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14002422988414764 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12268978357315063 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.027579162269830704 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 'particle/accuracy‚0.852745771408081 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18490751087665558 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2163204848766327 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19701999425888062 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32172951102256775 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007214151788502932 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10493827611207962 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3166666626930237 *jet/accuracy_3_of_6‚0.1538461595773697 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 % _timestamp‚1.7884077275314612e+09 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22604680061340332 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4234892427921295 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.40054449439048767 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9911898970603943 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666269302368 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5004450082778931 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9823604822158813 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13057804107666016 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5202518701553345 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989571213722229 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05859137326478958 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018820861354470253 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.39035090804100037 *jet/accuracy_2_of_4‚0.3435789942741394 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0595238097012043 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5177789926528931 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12444445490837097 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18494920432567596 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.332605242729187 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25196853280067444 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14797799289226532 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30557265877723694 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011640211567282677 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926407337188721 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3679388463497162 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1599099040031433 particle/accuracy_3_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12504589557647705 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4587453305721283 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1389341652393341 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5015089511871338 *"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_3‚0.0242901798337698 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867520332336426 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.42105263471603394 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15202409029006958 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959912300109863 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16090552508831024 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14797799289226532 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.47026345133781433 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18204934895038605 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47404929995536804 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_7_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10908859223127365 particle/accuracy_5_of_5‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2361110895872116 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 particle/accuracy_2_of_7‚1 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5004450082778931 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4188697338104248 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3363710939884186 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5015308856964111 *jet/accuracy_1_of_2‚0.7484655976295471 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4814908504486084 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49556559324264526 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45836105942726135 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17231008410453796 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 particle/accuracy_4_of_6‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9208037853240967 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,jet/accuracy_4_of_4‚0.003813933115452528 *jet/accuracy_2_of_5‚0.4425627291202545 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.40666666626930237 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6807581782341003 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03655914217233658 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5R"s‹Ü032641887664795 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.33050599694252014 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09945151954889297 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.48917344212532043 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1525862067937851 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 particle/accuracy_2_of_3‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7validation_average_jet_accuracy‚0.42899903655052185 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47185781598091125 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 jet/accuracy_4_of_7‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9892909526824951 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_4‚0.041674841195344925 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18842998147010803 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8967752456665039 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181723356247 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4440087378025055 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5125638246536255 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4969158172607422 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12622828781604767 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09831519424915314 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9974581599235535 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46924301981925964 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06637465953826904 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.117896169424057 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11974821239709854 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1361006796360016 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05029761791229248 )jet/accuracy_1_of_3‚0.851905107498169 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4097222089767456 +jet/accuracy_4_of_6‚0.03846153989434242 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.555573582649231 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10058891773223877 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873Öt»7lçí§ ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 particle/accuracy_4_of_6‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9208037853240967 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,jet/accuracy_4_of_4‚0.003813933115452528 *jet/accuracy_2_of_5‚0.4425627291202545 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.40666666626930237 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6807581782341003 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03655914217233658 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5032641887664795 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.33050599694252014 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09945151954889297 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.48917344212532043 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1525862067937851 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 particle/accuracy_2_of_3‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7validation_average_jet_accuracy‚0.42899903655052185 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47185781598091125 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 jet/accuracy_4_of_7‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9892909526824951 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_4‚0.041674841195344925 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18842998147010803 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8967752456665039 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181723356247 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4440087378025055 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5125638246536255 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4969158172607422 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12622828781604767 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09831519424915314 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9974581599235535 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46924301981925964 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06637465953826904 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.117896169424057 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11974821239709854 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1361006796360016 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05029761791229248 )jet/accuracy_1_of_3‚0.851905107498169 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4097222089767456 +jet/accuracy_4_of_6‚0.03846153989434242 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.555573582649231 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10058891773223877 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.289814829826355 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24112597107887268 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0672827661037445 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14174750447273254 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_6_of_6‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.05319149047136307 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 jet/accuracy_1_of_7‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996938169002533 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13996562361717224 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9887680411338806 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4818452298641205 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.495807021856308 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011659108102321625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08543571829795837 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12163742631673813 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36034226417541504 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.35327738523483276 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4973912537097931 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4388306438922882 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018674001097679138 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5216559171676636 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3777778148651123 *jet/accuracy_1_of_5‚0.9509536027908325 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9914838075637817 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.40689656138420105 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43845319747924805 particle/accuracy_5_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5179547667503357 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.44595959782600403 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4861111640930176 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4980917274951935 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4863683879375458 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022192273288965225 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_1_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.39506176114082336 particle/accuracy_1_of_6‚1 jet/accuracy_2_of_7‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9954047799110413 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25965818762779236 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4674477279186249 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2622280716896057 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4886071979999542 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46795985102653503 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23238733410835266 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_1_of_6‚0.9615384340286255 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.2966934144496918 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22420856356620789 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12969201803207397 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_7‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.500788688659668 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.48020488023757935 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 )loss/val_total_loss‚46.93182296504428 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4051200747489929 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9847952723503113 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3728395104408264 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022249136120080948 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04518524557352066 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5033667087554932 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49968722462654114 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1275961846113205 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.48936927318573 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5079364776611328 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024548934772610664 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3219163417816162 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9894781112670898 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15567342936992645 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_2_of_5‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39444440603256226 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333335638046265 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5063778758049011 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23339548707008362 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9843990206718445 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2083333432674408 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4600718319416046 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011749234981834888 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4276272654533386 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.49965283274650574 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1350667029619217 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_4_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32545432448387146 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25057467818260193 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4540540874004364 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _step‚296 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8064900636672974 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.531549870967865 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002858693478628993 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 jet/accuracy_5_of_6‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5083353519439697 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48306381702423096 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1149425357580185 jet/accuracy_3_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3701550364494324 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6364431381225586 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23695434629917145 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4696327745914459 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.061403512954711914 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17787349224090576 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1707923412322998 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5116744041442871 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3616279363632202 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.49407628178596497 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08137982338666916 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5030678510665894 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9907373189926147 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993182897567749 particle/accuracy_3_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 trainer/global_step‚3626 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9885729551315308 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20675250887870789 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4901207387447357 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.2448783963918686 (particle/f_score‚0.40554460883140564 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 _runtime‚ 2830.52152664 *jet/accuracy_4_of_5‚0.0098566310480237 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.4166666865348816 *jet/accuracy_2_of_6‚0.4038461446762085 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4597175717353821 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5648148059844971 particle/accuracy_6_of_7‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870175123214722 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20049557089805603 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905905723571777 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14223773777484894 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13843098282814026 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4030245244503021 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4246266186237335 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06321839243173599 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4437602460384369 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1377105414867401 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 epoch‚2 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4840024709701538 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9509766697883606 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 jet/accuracy_6_of_6‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14002422988414764 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12268978357315063 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.027579162269830704 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 'particle/accuracy‚0.852745771408081 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18490751087665558 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2163204848766327 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19701999425888062 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32172951102256775 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 >&Purity/1FRt0SRqqt2FBt/even´W¯Þ't_proportion‚0.01512896828353405 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007214151788502932 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10493827611207962 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3166666626930237 *jet/accuracy_3_of_6‚0.1538461595773697 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 % _timestamp‚1.7884077275314612e+09 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22604680061340332 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4234892427921295 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.40054449439048767 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9911898970603943 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666269302368 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5004450082778931 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9823604822158813 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13057804107666016 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5202518701553345 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989571213722229 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05859137326478958 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018820861354470253 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.39035090804100037 *jet/accuracy_2_of_4‚0.3435789942741394 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0595238097012043 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5177789926528931 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12444445490837097 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18494920432567596 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.332605242729187 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25196853280067444 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14797799289226532 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30557265877723694 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011640211567282677 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926407337188721 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3679388463497162 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1599099040031433 particle/accuracy_3_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12504589557647705 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4587453305721283 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1389341652393341 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5015089511871338 *"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_3‚0.0242901798337698 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867520332336426 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.42105263471603394 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15202409029006958 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959912300109863 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16090552508831024 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14797799289226532 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.47026345133781433 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18204934895038605 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47404929995536804 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_7_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10908859223127365 particle/accuracy_5_of_5‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2361110895872116 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 particle/accuracy_2_of_7‚1 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5004450082778931 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4188697338104248 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3363710939884186 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5015308856964111 *jet/accuracy_1_of_2‚0.7484655976295471 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4814908504486084 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49556559324264526 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45836105942726135 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17231008410453796 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0¨ËCY™è‚(:Ž ´ßãÔ’ý‡ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚32.0 gpu.0.powerWatts‚99.6' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.919999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aÎJé‚(:¿ ´ßãÔâÙè  network.sent‚ 7.832436e+06% disk./.usageGB‚236.71184921264648 disk.rbd1.in‚0 disk.drbd2265.in‚0* disk./.usagePercent‚15.885930815131266 disk.drbd1716.in‚0 disk.dm-1.out‚ 128.91015625 network.recv‚ 1.028066e+06 disk.sda2.out‚0# disk.nvme0c0n1.out‚ 7999.4140625 disk.drbd1716.out‚0 disk.rbd1.out‚14.65625 disk.sda.out‚ 128.91015625 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.dm-0.in‚ 16.6953125 disk.dm-0.out‚ 7999.4140625 disk.drbd2265.out‚0 disk.sda.in‚0 disk.drbd2266.in‚0# proc.memory.availableMB‚952.125 disk.rbd0.out‚0! disk.nvme0n1.out‚ 7999.4140625 disk.dm-1.in‚0 disk.sda3.out‚ 128.91015625 disk.drbd2080.in‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2819.88671875* proc.memory.percent‚17.211222648620605 cpu‚25.369222292288036 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 16.6953125$ memory_percent‚94.18869018554688œù?Wê‚(:’ ÃßãÔ÷ú‹ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚148.601' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.720200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0olPë‚(:Å ÃßãÔîŽë  disk.nvme0c0n1.in‚ 16.6953125 disk.rbd1.in‚0* proc.memory.percent‚17.211222648620605 proc.cpu.threads‚17 network.recv‚ 1.034057e+06 disk.sda.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚94.18973922729492 cpu‚25.367469559812175* disk./.usagePercent‚15.885930815131266% disk./.usageGB‚236.71184921264648 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 16.6953125 network.sent‚ 7.956755e+06 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 951.953125 disk.dm-1.in‚0 disk.dm-1.out‚ 129.52734375 disk.sda3.out‚ 129.52734375 disk.drbd2266.out‚0$ disk.nvme0c0n1.out‚ 7999.66015625 disk.rbd1.out‚14.65625" disk.nvme0n1.out‚ 7999.66015625 disk.dm-0.in‚ 16.6953125 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.dm-0.out‚ 7999.66015625# proc.memory.rssMB‚ 2819.88671875 disk.sda.out‚ 129.52734375–"}2ì- _wandbruntime‚2850  _runtime‚2850ÃjÀÈí ' lr-AdamW/pg1‚0.00024231960915204378 ' lr-AdamW/pg2‚0.00024231960915204378 trainer/global_step‚3649 $ _timestamp‚1.788407747964674e+09 _runtime‚2850.952105413 _step‚297ylƒÍîÇ ' lr-AdamW/pg1‚0.00024231960915204378 ' lr-AdamW/pg2‚0.00024231960915204378 trainer/global_step‚3649 $ _timestamp‚1.788407747964674e+09 _runtime‚2850.952105413 _step‚297©aá»72ï- _wandbruntime‚2850  _runtime‚2850²vÂàÝ'ð×O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122190429 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68982099 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3212903648768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207086280475648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3049501185378 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215727134540800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232204337477120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213079855616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚298 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3061429181952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3060772819456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53208330 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207086297515008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68982099 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3212903648768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207086280475648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3049501185378 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215727134540800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232204337477120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213079855616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚298 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3061429181952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3060772819456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53208330 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207086297515008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215784981386240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122222899 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31056324 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3213923349504 _step‚299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212722732763136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48641180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚ 2851.65011739 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53222346 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3061592247296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3213747155968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3062248623104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3062248623104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69020617 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48641639 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232266012154880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212722714760192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17585313 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210191502226550 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207141184606208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215784981386240 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69020617 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53222346 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232265988214784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3050972058306 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3061592247296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212722732763136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235479735370752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3050317620342 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69000553 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17584856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215784981386240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122222899 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31056324 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3213923349504 _step‚299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212722732763136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48641180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚ 2851.65011739 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53222346 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3061592247296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3213747155968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3062248623104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3062248623104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69020617 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48641639 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232266012154880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212722714760192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17585313 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210191502226550 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207141184606208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215784981386240 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69020617 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53222346 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232265988214784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3050972058306 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3061592247296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212722732763136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235479735370752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3050317620342 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69000553 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17584856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 Bë‚XöÒ /loss/FRt3/assignment_loss‚1.512236734231313 1loss/FBt2/assignment_loss‚0.25675855106156104 1loss/FBt3/assignment_loss‚0.18552771669598023 /loss/FRt3/detection_loss‚1.7140320065948698 -loss/FBt1/detection_loss‚3.76359423995018 1loss/FBt1/assignment_loss‚0.13251004039678568 trainer/global_step‚3649 _step‚300 /loss/FRt1/assignment_loss‚3.446481386820475 .loss/FBt4/detection_loss‚5.122564613819122 2loss/SRqqt1/assignment_loss‚2.1694261587279686 .loss/FRt1/detection_loss‚2.835532217314749 1loss/SRqqt1/detection_loss‚2.5648796830107186 1loss/SRqqt3/detection_loss‚1.0322793216942108 2loss/SRqqt2/assignment_loss‚1.6542004393413663 2loss/SRqqt3/assignment_loss‚1.1834882537847105 0loss/SRqqt4/detection_loss‚1.801322173564992 /loss/FBt2/detection_loss‚2.9034900102350445 _runtime‚ 2851.65163311 0loss/FRt2/assignment_loss‚3.5788156783327145 .loss/FRt2/detection_loss‚2.271484148629168 0loss/SRqqt2/detection_loss‚2.290776750817895 %loss/total_loss‚47.93772404981047 epoch‚3 0loss/FRt4/assignment_loss‚2.8442840636992943 /loss/FRt4/detection_loss‚2.5095858601283054 /loss/FBt3/detection_loss‚0.6518921740914145 % _timestamp‚1.7884077486642554e+09 1loss/SRqqt4/assignment_loss‚1.393081764591501 1loss/FBt4/assignment_loss‚0.11948006227612495-uß%]÷× 2loss/SRqqt2/assignment_loss‚1.6542004393413663 2loss/SRqqt3/assignment_loss‚1.1834882537847105 0loss/SRqqt4/detection_loss‚1.801322173564992 /loss/FBt2/detection_loss‚2.9034900102350445 _runtime‚ 2851.65163311 0loss/FRt2/assignment_loss‚3.5788156783327145 .loss/FRt2/detection_loss‚2.271484148629168 0loss/SRqqt2/detection_loss‚2.290776750817895 %loss/total_loss‚47.93772404981047 epoch‚3 0loss/FRt4/assignment_loss‚2.8442840636992943 /loss/FRt4/detection_loss‚2.5095858601283054 /loss/FBt3/detection_loss‚0.6518921740914145 % _timestamp‚1.7884077486642554e+09 1loss/SRqqt4/assignment_loss‚1.393081764591501 1loss/FBt4/assignment_loss‚0.11948006227612495 /loss/FRt3/assignment_loss‚1.512236734231313 1loss/FBt2/assignment_loss‚0.25675855106156104 1loss/FBt3/assignment_loss‚0.18552771669598023 /loss/FRt3/detection_loss‚1.7140320065948698 -loss/FBt1/detection_loss‚3.76359423995018 1loss/FBt1/assignment_loss‚0.13251004039678568 trainer/global_step‚3649 _step‚300 /loss/FRt1/assignment_loss‚3.446481386820475 .loss/FBt4/detection_loss‚5.122564613819122 2loss/SRqqt1/assignment_loss‚2.1694261587279686 .loss/FRt1/detection_loss‚2.835532217314749 1loss/SRqqt1/detection_loss‚2.5648796830107186 1loss/SRqqt3/detection_loss‚1.0322793216942108¬ÉoÜúø‚(:„ ÒßãÔÚë¡ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.23 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Vp¾Kù‚(:À ÒßãÔÉ¡–  disk.nvme0n1.in‚ 16.6953125 disk.drbd2080.out‚0$ memory_percent‚94.19000148773193# proc.memory.rssMB‚ 2819.88671875 disk.drbd2265.out‚0* disk./.usagePercent‚15.885808955544967 disk.rbd0.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd1716.in‚0 disk.dm-0.in‚ 16.6953125 disk.drbd2266.out‚0 network.sent‚ 7.962779e+06 disk.rbd1.out‚14.8125 disk.dm-1.in‚0 disk.dm-1.out‚ 129.9140625 disk.drbd2265.in‚0 disk.sda3.out‚ 129.9140625 disk.drbd2266.in‚0 proc.cpu.threads‚17 network.recv‚ 1.038476e+06 disk.sda.out‚ 129.9140625* proc.memory.percent‚17.211222648620605 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚236.71003341674805 disk.rbd0.in‚0 disk.sda2.in‚0! disk.nvme0n1.out‚ 8473.2890625 disk.dm-0.out‚ 8473.2890625 cpu‚25.365815557750683 disk.drbd2080.in‚0( proc.memory.availableMB‚ 951.91015625# disk.nvme0c0n1.out‚ 8473.2890625 disk.drbd1716.out‚0 disk.rbd1.in‚0ä{2§’ú‚(:‡ áßãÔ‡‚’ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ˜Æ·Qû‚(:Æ áßãÔ‰ªì  disk.dm-0.out‚ 8473.54296875 disk.nvme0c0n1.in‚ 16.6953125# proc.memory.rssMB‚ 2819.88671875 disk.nvme0n1.in‚ 16.6953125 disk.sda.out‚ 130.52734375 disk.drbd2080.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚17.211222648620605 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.dm-0.in‚ 16.6953125 disk.dm-1.in‚0 disk.dm-1.out‚ 130.52734375 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.out‚ 8473.54296875 disk.drbd2266.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 8473.54296875 disk.drbd2265.out‚0 network.recv‚ 1.045263e+06% disk./.usageGB‚236.70997619628906 disk.rbd0.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 951.66796875 cpu‚25.364190802887137 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚94.19147968292236* disk./.usagePercent‚15.885805115431953 disk.sda3.out‚ 130.52734375 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚14.8125 network.sent‚ 8.071807e+06–‚¢•2ü- _wandbruntime‚2886  _runtime‚2886 Š²Éýà trainer/global_step‚3699 % _timestamp‚1.7884077839627368e+09 _runtime‚2886.950166432 _step‚301 ' lr-AdamW/pg1‚0.00024102291240273618 ' lr-AdamW/pg2‚0.00024102291240273618–ئÎþÈ ' lr-AdamW/pg1‚0.00024102291240273618 ' lr-AdamW/pg2‚0.00024102291240273618 trainer/global_step‚3699 % _timestamp‚1.7884077839627368e+09 _runtime‚2886.950166432 _step‚301­âõ;ß2ÿ- _wandbruntime‚2886  _runtime‚2886‘Ý™=€×O YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215484013663744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238542785055744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209831487003648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53909132 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚218586446654464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53909132 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚3699 _runtime‚2886.951029581 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218585772289024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49260887 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235286874360320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49261316 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238542595394560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235286898300416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215484031666688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3101758625280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69928189{€Ÿmâ'ÜO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚218585772289024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17804331 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3255721034240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3090990962542 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212922495005550 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123837321 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3101758625280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69908151 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3090336524786 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123817281 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3255886755328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69928189 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53909130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31456558 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17804758 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218585772289024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49260887 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235286874360320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49261316 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238542595394560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235286898300416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215484031666688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3101758625280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69928189 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215484013663744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238542785055744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209831487003648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53909132 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚218586446654464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53909132 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚3699 _runtime‚2886.951029581 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚2887.623097912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49273570 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69946669 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123869817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49274001 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17809275 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209886391134208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218643619134464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215540384705536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3091152959750 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235348549038080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3102578053120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215540366702592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3091807397714 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69926605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3256584464384=œ0NB„æM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31464726 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884077846353283e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3102578053120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215540366702592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3091807397714 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69926605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3256584464384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚2887.623097912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49273570 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69946669 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123869817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49274001 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17809275 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209886391134208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218643619134464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215540384705536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3091152959750 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235348549038080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54609932 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3143400793600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215707865395842 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70854241 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54609932 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚218301665606144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70834203 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3142744431104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54609930 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218301683609088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241668098789888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212576693531648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31864956 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18030655 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221445084402688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125464173 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238369411243520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54609930 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221444410037248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221444410037248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54609932 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3143400793600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215707865395842 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70854241 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54609932 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚218301665606144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70834203 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3142744431104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54609930 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218301683609088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241668098789888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212576693531648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31864956 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18030655 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221445084402688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125464173 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238369411243520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54609930 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884078208701427e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 1254966]çò¨†69 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3131988299158 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215763585961366 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚238431085921280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221501582503936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125476603 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241730408731136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212631597662208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70852657 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299512457728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125496669 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚307 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221502256882688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221502256882688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31873126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18035083 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70872721 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218358036647936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚218358018644992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241730598379008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215764257438690 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218358036647936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54623946 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218358018644992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70872721 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3132642737122 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3143563858944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31873124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49907766 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844—¢¥²ì&”æM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221502256882688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221502256882688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31873126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18035083 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70872721 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218358036647936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚218358018644992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241730598379008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215764257438690 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218358036647936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54623946 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218358018644992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70872721 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3132642737122 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3143563858944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31873124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49907766 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3144220234752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884078208701427e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125496669 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3131988299158 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215763585961366 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚238431085921280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221501582503936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125476603 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241730408731136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212631597662208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70852657 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299512457728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125496669 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚307³åI¹˜2•- _wandbruntime‚2923  _runtime‚2923º²ÁY–Ó /loss/FBt1/detection_loss‚2.9261347204446793 epoch‚3 0loss/FRt2/assignment_loss‚3.7211320924758913 2loss/SRqqt1/assignment_loss‚1.9227154275466656 2loss/SRqqt3/assignment_loss‚0.9135508010945013 % _timestamp‚1.7884078208717496e+09 _step‚308 0loss/FRt1/assignment_loss‚3.8300995588302613 2loss/SRqqt2/assignment_loss‚1.7608040865172039 .loss/FRt3/detection_loss‚1.976034813137217 1loss/SRqqt1/detection_loss‚2.5467531475527534 %loss/total_loss‚45.41949809750557 trainer/global_step‚3749 1loss/FBt3/assignment_loss‚0.19569861740712025 1loss/SRqqt3/detection_loss‚1.0283015292858886 /loss/FRt3/assignment_loss‚1.255817181684754 -loss/FRt4/detection_loss‚2.47059195737044 0loss/FBt4/assignment_loss‚0.6431278325617313 1loss/SRqqt2/detection_loss‚1.5764940985224463 .loss/FBt3/detection_loss‚0.669807134185579 _runtime‚2923.859096675 1loss/SRqqt4/assignment_loss‚1.615628579976382 /loss/FBt2/assignment_loss‚0.513681021984666 /loss/FRt1/detection_loss‚2.7193743482232096 /loss/FRt2/detection_loss‚2.3253099691867827 /loss/FBt2/detection_loss‚2.9073340197404227 .loss/FBt4/detection_loss‚3.041319638490677 0loss/FRt4/assignment_loss‚2.9486053614389327 1loss/FBt1/assignment_loss‚0.11124362928967457 1loss/SRqqt4/detection_loss‚1.7999385305576854=¾Ûf^—Ø /loss/FBt2/detection_loss‚2.9073340197404227 .loss/FBt4/detection_loss‚3.041319638490677 0loss/FRt4/assignment_loss‚2.9486053614389327 1loss/FBt1/assignment_loss‚0.11124362928967457 1loss/SRqqt4/detection_loss‚1.7999385305576854 /loss/FBt1/detection_loss‚2.9261347204446793 epoch‚3 0loss/FRt2/assignment_loss‚3.7211320924758913 2loss/SRqqt1/assignment_loss‚1.9227154275466656 2loss/SRqqt3/assignment_loss‚0.9135508010945013 % _timestamp‚1.7884078208717496e+09 _step‚308 0loss/FRt1/assignment_loss‚3.8300995588302613 2loss/SRqqt2/assignment_loss‚1.7608040865172039 .loss/FRt3/detection_loss‚1.976034813137217 1loss/SRqqt1/detection_loss‚2.5467531475527534 %loss/total_loss‚45.41949809750557 trainer/global_step‚3749 1loss/FBt3/assignment_loss‚0.19569861740712025 1loss/SRqqt3/detection_loss‚1.0283015292858886 /loss/FRt3/assignment_loss‚1.255817181684754 -loss/FRt4/detection_loss‚2.47059195737044 0loss/FBt4/assignment_loss‚0.6431278325617313 1loss/SRqqt2/detection_loss‚1.5764940985224463 .loss/FBt3/detection_loss‚0.669807134185579 _runtime‚2923.859096675 1loss/SRqqt4/assignment_loss‚1.615628579976382 /loss/FBt2/assignment_loss‚0.513681021984666 /loss/FRt1/detection_loss‚2.7193743482232096 /loss/FRt2/detection_loss‚2.3253099691867827´: ”é’˜‚(:‡ ŽàãÔ‹’“ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 á=÷Z™‚(:Ï ŽàãÔÅØÍ  disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2720.13671875% disk./.usageGB‚236.70997619628906 disk.dm-1.out‚ 132.015625 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 16.6953125' disk.nvme0c0n1.out‚8474.99658203125 disk.nvme0n1.in‚ 16.6953125# memory_percent‚93.6244249343872" disk.dm-0.out‚8474.99658203125 disk.sda.in‚0 disk.sda3.in‚0% disk.nvme0n1.out‚8474.99658203125 proc.cpu.threads‚17 network.recv‚ 1.059091e+06 disk.sda.out‚ 132.015625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 15.0009765625) proc.memory.availableMB‚ 1044.57421875 disk.dm-0.in‚ 16.6953125 disk.rbd1.in‚0 cpu‚25.359317695531093 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.602396965026855 network.sent‚ 8.191864e+06* disk./.usagePercent‚15.885805115431953 disk.drbd2265.in‚0 disk.drbd1716.out‚0 disk.sda3.out‚ 132.015625JX’š‚(:‡ àãÔÄÇù gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.1238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òRh/N›‚(:à àãÔ³‚Þ  disk.nvme0c0n1.in‚ 16.6953125 disk.dm-0.in‚ 16.6953125 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2720.13671875 disk.drbd2266.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 network.recv‚ 1.065451e+06 disk.nvme0n1.in‚ 16.6953125 disk.drbd2080.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.602396965026855 disk.drbd2266.in‚0$ memory_percent‚93.62325668334961) disk./.usagePercent‚15.88581330767305 disk.drbd1716.out‚0" disk.nvme0n1.out‚ 8475.46484375 disk.dm-0.out‚ 8475.46484375 disk.dm-1.out‚ 132.671875 disk.sda.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 1044.765625% disk./.usageGB‚236.71009826660156 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 8475.46484375 disk.rbd1.in‚0 disk.sda3.out‚ 132.671875 disk.drbd2080.out‚0 disk.rbd1.out‚ 15.0869140625 disk.dm-1.in‚0 disk.sda.out‚ 132.671875 cpu‚25.35783488197644 network.sent‚ 8.300679e+06 disk.sda1.out‚0$½þœ‚(:’ ¬àãÔåæ¢ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚142.643' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.528599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÎ×ËV‚(:Ë ¬àãÔñ†  disk.sda.out‚ 133.109375 disk.drbd2080.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2720.13671875" disk.dm-0.out‚8476.20751953125 network.sent‚ 8.306187e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.rbd1.in‚0 disk.dm-1.out‚ 133.109375 disk.rbd0.in‚0 disk.rbd1.out‚ 15.2353515625% disk.nvme0n1.out‚8476.20751953125 disk.dm-0.in‚ 16.6953125 disk.drbd2265.out‚0' proc.memory.availableMB‚ 570.1015625* proc.memory.percent‚16.602396965026855 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚8476.20751953125$ memory_percent‚96.52037620544434 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda3.out‚ 133.109375 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 network.recv‚ 1.069757e+06) disk./.usagePercent‚15.88581151562031$ disk./.usageGB‚236.7100715637207 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 16.6953125 cpu‚25.356195829669020 ¡â2ž- _wandbruntime‚2959  _runtime‚2959“FÓÕÉŸÃ ' lr-AdamW/pg2‚0.00023836509002798852 trainer/global_step‚3799 % _timestamp‚1.7884078565337882e+09 _runtime‚2959.521230222 _step‚309 ' lr-AdamW/pg1‚0.00023836509002798852YT .Î È ' lr-AdamW/pg1‚0.00023836509002798852 ' lr-AdamW/pg2‚0.00023836509002798852 trainer/global_step‚3799 % _timestamp‚1.7884078565337882e+09 _runtime‚2959.521230222 _step‚309µÄ+2¡- _wandbruntime‚2959  _runtime‚2959R¬ƒ{¢ÖO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224303047785472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18256604 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3172661641358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172007203602 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127070985 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224303722150912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3184386599424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127091025 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244793543762432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3341395488768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224303722150912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241451948126720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55310730 _step‚310 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71780293 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3183730236928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215321917099008 _runtime‚2959.522164084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55310732 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224303047785472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50529960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55310732 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218494578740366 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3341571695616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_#ÑÔÈatrain_batch_start/allocated_bytes.large_pool.allocated‚221119335551488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241451972066816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 $ _timestamp‚1.788407856534474e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50530410 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18257052 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218493907263250 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221119317548544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414’m…á'£ÛO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55310732 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218494578740366 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3341571695616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221119335551488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241451972066816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 $ _timestamp‚1.788407856534474e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50530410 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18257052 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218493907263250 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221119317548544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224303047785472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18256604 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3172661641358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172007203602 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127070985 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224303722150912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3184386599424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127091025 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244793543762432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3341395488768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224303722150912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241451948126720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55310730 _step‚310 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71780293 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3183730236928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215321917099008 _runtime‚2959.522164084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55310732 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224303047785472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50529960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944¶ÿÙe%2¤- _wandbruntime‚2960  _runtime‚2960a^ãQå&¥ßM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3184549664768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32281524 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55324748 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3342413927936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3342239831552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3173478076530 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3184549664768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241513598864384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127103455 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚ 2960.20219339 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71798773 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218549627828774 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71778709 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚224360220252160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55324748 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3185206040576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3185206040576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50542646 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71798773 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221175670587392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18261122 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221175670587392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241513622804480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215376821229568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788407857214151e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚311 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127103455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3172823638566 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55324746 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221175688590336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚3799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55324746 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18261555 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218550299306098 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127123521 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71778709 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0±W­\ê&¦äM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241513598864384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127103455 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚ 2960.20219339 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71798773 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218549627828774 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71778709 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚224360220252160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55324748 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3185206040576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3185206040576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50542646 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71798773 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221175670587392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18261122 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221175670587392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241513622804480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215376821229568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788407857214151e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚311 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127103455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3172823638566 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55324746 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221175688590336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚3799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55324746 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18261555 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218550299306098 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127123521 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71778709 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3184549664768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32281524 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55324748 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3342413927936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3342239831552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3173478076530 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3184549664768·‹®üo2§- _wandbruntime‚2960  _runtime‚2960š´nðZ¨Ô % _timestamp‚1.7884078572158494e+09 2loss/SRqqt4/assignment_loss‚1.8059239565261773 %loss/total_loss‚46.20176807116804 epoch‚3 0loss/FRt1/assignment_loss‚3.3857025589261736 0loss/FBt2/assignment_loss‚0.5051931427500677 /loss/FRt1/detection_loss‚2.8585186788014005 1loss/SRqqt1/detection_loss‚2.42102428442907ÞÂ204 1loss/SRqqt4/detection_loss‚1.5210458446826254 1loss/SRqqt1/assignment_loss‚2.099108246465524 .loss/FRt2/detection_loss‚2.232294894590059 .loss/FRt4/detection_loss‚2.134564032709157 0loss/SRqqt2/detection_loss‚2.182725490355978 1loss/FBt3/assignment_loss‚0.21852193977794937 /loss/FRt3/detection_loss‚2.4590802094766073 _runtime‚2960.203198516 _step‚312 .loss/FRt4/assignment_loss‚2.86073985475081 1loss/FBt4/assignment_loss‚0.07936902903020382 1loss/SRqqt3/detection_loss‚1.0544215098596537 /loss/FBt4/detection_loss‚3.0744556983311973 trainer/global_step‚3799 0loss/FRt3/assignment_loss‚1.3956158229282924 /loss/FBt1/detection_loss‚2.7571192582448325 /loss/FBt2/detection_loss‚2.6643967509269713 /loss/FRt2/assignment_loss‚4.297458706961738 1loss/SRqqt2/assignment_loss‚1.954014458370452 2loss/SRqqt3/assignment_loss‚1.1970294853242545 /loss/FBt3/detection_loss‚0.7381576247658813 1loss/FBt1/assignment_loss‚0.30528659218301374›H_©Ù 1loss/FBt3/assignment_loss‚0.21852193977794937 /loss/FRt3/detection_loss‚2.4590802094766073 _runtime‚2960.203198516 _step‚312 .loss/FRt4/assignment_loss‚2.86073985475081 1loss/FBt4/assignment_loss‚0.07936902903020382 1loss/SRqqt3/detection_loss‚1.0544215098596537 /loss/FBt4/detection_loss‚3.0744556983311973 trainer/global_step‚3799 0loss/FRt3/assignment_loss‚1.3956158229282924 /loss/FBt1/detection_loss‚2.7571192582448325 /loss/FBt2/detection_loss‚2.6643967509269713 /loss/FRt2/assignment_loss‚4.297458706961738 1loss/SRqqt2/assignment_loss‚1.954014458370452 2loss/SRqqt3/assignment_loss‚1.1970294853242545 /loss/FBt3/detection_loss‚0.7381576247658813 1loss/FBt1/assignment_loss‚0.30528659218301374 % _timestamp‚1.7884078572158494e+09 2loss/SRqqt4/assignment_loss‚1.8059239565261773 %loss/total_loss‚46.20176807116804 epoch‚3 0loss/FRt1/assignment_loss‚3.3857025589261736 0loss/FBt2/assignment_loss‚0.5051931427500677 /loss/FRt1/detection_loss‚2.8585186788014005 1loss/SRqqt1/detection_loss‚2.4210242844290204 1loss/SRqqt4/detection_loss‚1.5210458446826254 1loss/SRqqt1/assignment_loss‚2.099108246465524 .loss/FRt2/detection_loss‚2.232294894590059 .loss/FRt4/detection_loss‚2.134564032709157 0loss/SRqqt2/detection_loss‚2.182725490355978¸ÿ)‘ª‚(:’ »àãÔß´® gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.288' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.257600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v=ß×^«‚(:Ó »àãÔß‘ù  disk.sda2.in‚0 network.sent‚ 8.31211e+06 disk.nvme0n1.in‚ 16.6953125% disk.nvme0n1.out‚8476.61474609375" disk.dm-0.out‚8476.61474609375 disk.sda.in‚0% disk./.usageGB‚236.71051406860352 disk.drbd2265.out‚0' disk.nvme0c0n1.out‚8476.61474609375 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2720.20703125 cpu‚25.354664227310778 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.sda3.out‚ 133.73046875 disk.rbd0.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 570.44140625 disk.sda.out‚ 133.73046875 disk.rbd0.in‚0 disk.sda1.out‚0 network.recv‚ 1.074014e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 15.2353515625* disk./.usagePercent‚15.885841212494281 disk.drbd2266.out‚0$ memory_percent‚96.51830196380615) proc.memory.percent‚16.60282611846924 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125 disk.dm-1.in‚0 disk.dm-1.out‚ 133.73046875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚ 16.6953125õŠ~’¬‚(:‡ ÊàãÔ€¢œ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.098' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ŠÒW­‚(:Ì ÊàãÔÙ‚ ! disk.dm-0.out‚8477.2685546875 disk.drbd2265.in‚0( proc.memory.availableMB‚ 569.95703125 cpu‚25.353079229235576& disk.nvme0c0n1.out‚8477.2685546875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125 disk.sda.in‚0 disk.sda1.in‚0) proc.memory.percent‚16.60282611846924$ disk.nvme0n1.out‚8477.2685546875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.sent‚ 8.419782e+06 network.recv‚ 1.07974e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 134.1953125 disk.sda3.out‚ 134.1953125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd2265.out‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2720.20703125% disk./.usageGB‚236.71050262451172 disk.rbd1.in‚0 disk.nvme0n1.in‚ 16.6953125 disk.drbd2080.out‚0 disk.sda.out‚ 134.1953125) disk./.usagePercent‚15.88584044447168 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚96.52125835418701 proc.cpu.threads‚17 disk.rbd1.out‚ 15.3916015625÷[.Ñ2®- _wandbruntime‚2995  _runtime‚2995&ð¶ÇƯÀ $ _timestamp‚1.788407892849974e+09 _runtime‚2995.837422202 _step‚313 & lr-AdamW/pg1‚0.0002370045182887477 & lr-AdamW/pg2‚0.0002370045182887477 trainer/global_step‚3849ö¥­éË°Å trainer/global_step‚3849 $ _timestamp‚1.788407892849974e+09 _runtime‚2995.837422202 _step‚313 & lr-AdamW/pg1‚0.0002370045182887477 & lr-AdamW/pg2‚0.0002370045182887477¹o+ò2±- _wandbruntime‚2995  _runtime‚2995õ¤AÝ'²×O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221280620607774 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247918856526336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3213496980766 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 % _timestamp‚1.7884078928508158e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72706345 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72706345 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218067123627008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128697837 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56011532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3384347576320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚3849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3225372405248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223936987493888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227161685533696 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128717877 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3225372405248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227162359899136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3224716042752 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218067106587648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3384186049536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18483729 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247918671059456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227161685533696 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2995.838504394 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32681758 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72686307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18484163 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51165921 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227162359899136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 % _timestamp‚1.7884078928508158e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72706345 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72706345 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218067123627008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128697837 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56011532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3384347576320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚3849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3225372405248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223936987493888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227161685533696 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128717877 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3225372405248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227162359899136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3224716042752 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218067106587648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3384186049536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18483729 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247918671059456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227161685533696 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2995.838504394 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32681758 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72686307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18484163 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51165921 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227162359899136 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225535470592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51178592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3385040184832 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128750373 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227219532379136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚2996.511641584 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218122010718208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223993340532736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72704761 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56025546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3226191846400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128730307 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223993322529792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56025546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32689924 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18488220 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221335669696182 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227218858000384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213658977974 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223993340532736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56025548 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227219532379136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244596159687680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51178144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32689926 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ЬŠÙë&¶åM trainer/global_step‚3849 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3225535470592 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213658977974 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223993340532736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56025548 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227219532379136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244596159687680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51178144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32689926 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225535470592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51178592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3385040184832 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128750373 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227219532379136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚2996.511641584 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218122010718208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223993340532736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72704761 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56025546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3226191846400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128730307 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223993322529792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56025546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32689924 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18488220 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221335669696182 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227218858000384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230020323281920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3266358211072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230020997647360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247617045833216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73632397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56712332 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51801529 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33090156 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226754639436288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3427365263872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3265701848576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226754639436288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56712330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3427197445632³Ï‚Þ´ÅÛO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3265701848576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226754639436288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56712330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 _(—¢K-$MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3427197445632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224066662475182 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56712332 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230020323281920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3266358211072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230020997647360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247617045833216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73632397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56712332 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51801529 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33090156 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226754639436288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3427365263872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¾*Ô'S2Æ- _wandbruntime‚3032  _runtime‚3032ä‹‘ôç&ÇáM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73630813 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56726348 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224121711563590 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚3032.823684011 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226810992475136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56726346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130357159 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220867217246208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226810974472192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3266521276416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230078170127360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18715869 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130357159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51813733 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3267177652224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226810974472192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18715409 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56726348 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247678672630784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230077495748608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224122383040914 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230078170127360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226810992475136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130377225 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3267177652224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3428219130880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884079298358595e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247678696570880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3428045034496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73650877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33098324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251106717665280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚319 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56726346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3255148755346 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33098326 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56726348 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224121711563590 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚3032.823684011 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226810992475136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56726346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130357159 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220867217246208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226810974472192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3266521276416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230078170127360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18715869 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130357159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51813733 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3267177652224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226810974472192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18715409 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56726348 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247678672630784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230077495748608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224122383040914 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230078170127360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226810992475136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130377225 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3267177652224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3428219130880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884079298358595e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247678696570880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3428045034496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73650877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33098324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251106717665280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚319 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56726346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3255148755346 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33098326 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E2Ð- _wandbruntime‚3068  _runtime‚3068Al¯ªÉÑà ' lr-AdamW/pg1‚0.00023422147135299844 ' lr-AdamW/pg2‚0.00023422147135299844 trainer/global_step‚3949 % _timestamp‚1.7884079656060038e+09 _runtime‚3068.593483561 _step‚321$X~…ÎÒÈ _step‚321 ' lr-AdamW/pg1‚0.00023422147135299844 ' lr-AdamW/pg2‚0.00023422147135299844 trainer/global_step‚3949 % _timestamp‚1.7884079656060038e+09 _runtime‚3068.593483561Á‹«§S2Ó- _wandbruntime‚3068  _runtime‚3068³“ûÂÝ'Ô×O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33498556 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250699558776320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3307344016896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74558449 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3470094315520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3470251648000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3295167659582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18937961 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229572291378688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57413132 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33498558 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57413130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223557519643648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232879635395584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229572273375744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250699582716416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131971581 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254169834364416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57413132 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226852704342590 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232878961030144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131951541 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57413130 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232878961030144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3306687654400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232879635395584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229572273375744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3307344016896 _step‚322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52436517 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74558449 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3470251648000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3295167659582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18937961 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229572291378688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57413132 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33498558 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57413130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223557519643648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232879635395584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229572273375744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250699582716416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131971581 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254169834364416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57413132 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226852704342590 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232878961030144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131951541 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57413130 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232878961030144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3306687654400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232879635395584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229572273375744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3307344016896 _step‚322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52436517 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74558449 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33498556 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250699558776320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3307344016896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025ÂÇ9lž2Ö- _wandbruntime‚3069  _runtime‚3069Y–±š$×áM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229628644417536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884079662901654e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226908424908322 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3307507082240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74556865 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223612423774208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3307507082240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33506724 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132004077 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57427146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254232162077184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3295984094754 _step‚323 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3308163458048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3295329656790 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74576929 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74556865 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229628626414592 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232936807875584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57427148 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18942508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3470952563200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131984011 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223612440813568 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229628626414592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3471116173824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250761233454080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229628644417536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57427146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52449679 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3308163458048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52449232 E?DeviceStatsMonitor.on_train_batch_eÂÒ~Mnd/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚3069.278025856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33506726 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18942953 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254232349627904 trainer/global_step‚3949r_r4ì&ØæM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232936807875584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57427148 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18942508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3470952563200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131984011 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223612440813568 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229628626414592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3471116173824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250761233454080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229628644417536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57427146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52449679 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3308163458048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52449232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚3069.278025856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33506726 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18942953 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254232349627904 trainer/global_step‚3949 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229628644417536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884079662901654e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226908424908322 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3307507082240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74556865 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223612423774208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3307507082240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33506724 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132004077 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57427146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254232162077184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3295984094754 _step‚323 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3308163458048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3295329656790 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74576929 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74556865 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229628626414592 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025Ã"•á%2Ù- _wandbruntime‚3069  _runtime‚3069G™ZÚÔ trainer/global_step‚3949 0loss/FRt4/assignment_loss‚3.2920019111850043 2loss/SRqqt2/assignment_loss‚1.8379759268462659 0loss/FBt3/assignment_loss‚0.3087069695416095 /loss/FRt1/detection_loss‚3.0773792730437384 0loss/SRqqt1/detection_loss‚2.431084733456373 0loss/FBt1/assignment_loss‚0.4999402946096476 /loss/FRt3/detection_loss‚1.7298625403759527 .loss/FBt2/detection_loss‚2.526807983716329 %loss/total_loss‚46.74086916553334 epoch‚3 2loss/SRqqt1/assignment_loss‚2.1981100379489362 2loss/SRqqt4/assignment_loss‚1.9636574431788176 0loss/SRqqt2/detection_loss‚1.902307488322258 % _timestamp‚1.7884079662918112e+09 /loss/FRt2/detection_loss‚2.4821865310271582 2loss/SRqqt3/assignment_loss‚0.8612779598931471 0loss/SRqqt3/detection_loss‚0.832824474821488 _runtime‚3069.279187519 0loss/FRt1/assignment_loss‚3.9802370601230197 0loss/FBt2/assignment_loss‚0.4875925276428461 0loss/FBt4/assignment_loss‚0.4665880545973778 /loss/FBt3/detection_loss‚0.5780898031054271 _step‚324 0loss/FRt2/assignment_loss‚3.5921661243552254 0loss/FRt3/assignment_loss‚1.3427424290600944 /loss/FRt4/detection_loss‚2.3556734696030617 1loss/SRqqt4/detection_loss‚1.9312198345238965 .loss/FBt1/detection_loss‚2.703445649147034 /loss/FBt4/detection_loss‚3.3589906454086305Ð €Í_ÛÙ 0loss/FBt1/assignment_loss‚0.4999402946096476 /loss/FRt3/detection_loss‚1.7298625403759527 .loss/FBt2/detection_loss‚2.526807983716329 %loss/total_loss‚46.74086916553334 epoch‚3 2loss/SRqqt1/assignment_loss‚2.1981100379489362 2loss/SRqqt4/assignment_loss‚1.9636574431788176 0loss/SRqqt2/detection_loss‚1.902307488322258 % _timestamp‚1.7884079662918112e+09 /loss/FRt2/detection_loss‚2.4821865310271582 2loss/SRqqt3/assignment_loss‚0.8612779598931471 0loss/SRqqt3/detection_loss‚0.832824474821488 _runtime‚3069.279187519 0loss/FRt1/assignment_loss‚3.9802370601230197 0loss/FBt2/assignment_loss‚0.4875925276428461 0loss/FBt4/assignment_loss‚0.4665880545973778 /loss/FBt3/detection_loss‚0.5780898031054271 _step‚324 0loss/FRt2/assignment_loss‚3.5921661243552254 0loss/FRt3/assignment_loss‚1.3427424290600944 /loss/FRt4/detection_loss‚2.3556734696030617 1loss/SRqqt4/detection_loss‚1.9312198345238965 .loss/FBt1/detection_loss‚2.703445649147034 /loss/FBt4/detection_loss‚3.3589906454086305 trainer/global_step‚3949 0loss/FRt4/assignment_loss‚3.2920019111850043 2loss/SRqqt2/assignment_loss‚1.8379759268462659 0loss/FBt3/assignment_loss‚0.3087069695416095 /loss/FRt1/detection_loss‚3.0773792730437384 0loss/SRqqt1/detection_loss‚2.431084733456373Äφ|IÜ‚(:’ ¤áãÔ½¸ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.529' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.105799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TF¤UÝ‚(:Ê ¤áãÔåê   cpu‚25.344178469211155 disk.dm-1.out‚ 139.59765625 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0) proc.memory.percent‚16.60282611846924 disk.rbd1.out‚ 15.6123046875 disk.nvme0n1.in‚ 16.6953125 disk.sda2.in‚0 disk.sda3.out‚ 139.59765625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2080.in‚0' proc.memory.availableMB‚ 570.3203125 network.recv‚ 1.108187e+06% disk./.usageGB‚235.38269805908203" disk.nvme0n1.out‚ 8484.62109375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd2266.out‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 8484.62109375# proc.memory.rssMB‚ 2720.20703125 disk.sda1.out‚0 disk.dm-1.in‚0 network.sent‚ 8.660323e+06 disk.dm-0.out‚ 8484.62109375 proc.cpu.threads‚17* disk./.usagePercent‚15.796730365983455 disk.rbd1.in‚0 disk.sda.out‚ 139.59765625$ memory_percent‚96.519041061401371@× Þ‚(:’ ³áãÔÇ¿ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.549' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.909800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÐ%ºYß‚(:Î ³áãÔçüÍ  disk.drbd1716.in‚0$ disk.nvme0n1.out‚8485.3525390625 disk.drbd2265.in‚0 cpu‚25.342692657119606 proc.cpu.threads‚17 disk.rbd1.out‚ 15.6123046875# proc.memory.rssMB‚ 2720.20703125% disk./.usageGB‚235.38272094726562! disk.dm-0.out‚8485.3525390625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚15.796731902028661 disk.rbd0.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 570.06640625 network.recv‚ 1.114326e+06 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 140.7265625$ memory_percent‚96.52059078216553) proc.memory.percent‚16.60282611846924 network.sent‚ 8.769034e+06 disk.sda1.in‚0& disk.nvme0c0n1.out‚8485.3525390625 disk.nvme0n1.in‚ 16.6953125 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 140.7265625 disk.sda3.out‚ 140.7265625Nc‚2à- _wandbruntime‚3104  _runtime‚3104 ʃþÇáÁ & lr-AdamW/pg1‚0.0002327995761393001 & lr-AdamW/pg2‚0.0002327995761393001 trainer/global_step‚3999 % _timestamp‚1.7884080017992175e+09 _runtime‚3104.786657957 _step‚325ÖÅ6ÌâÆ % _timestamp‚1.7884080017992175e+09 _runtime‚3104.786657957 _step‚325 & lr-AdamW/pg1‚0.0002327995761393001 & lr-AdamW/pg2‚0.0002327995761393001 trainer/global_step‚3999Å:M2ã- _wandbruntime‚3104  _runtime‚3104ÝDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232389925318144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226302743211008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226302726171648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133598433 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚326 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235737598778368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33906958 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253782119599616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3347673460224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3335348561234 trainer/global_step‚3999 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257295239930880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚257295046075392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3513120331264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53071329 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58113932 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53070874 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33906956 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133578393 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58113932ÇŒ£·åÜO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232389925318144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226302743211008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226302726171648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133598433 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚326 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235737598778368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33906958 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253782119599616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3347673460224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3335348561234 trainer/global_step‚3999 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257295239930880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚257295046075392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3513120331264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53071329 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58113932 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53070874 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33906956 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133578393 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58113932 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58113930 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75484501 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235737598778368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3512950415872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3336002998990 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75464463 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232389943321088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229638074732882 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884080017999308e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3347673460224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232389925318144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19164371 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75464463 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19163918 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253782095659520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232389943321088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928Æá æ‚(:… ÂáãÔ†›ˆ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.02' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^SüŒXç‚(:Í ÂáãÔÍü¬  disk.drbd2266.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.119031e+06 disk.sda.out‚ 141.171875 disk.rbd0.in‚0 disk.dm-0.in‚ 16.6953125 disk.rbd0.out‚0 disk.rbd1.out‚ 15.7919921875 disk.sda3.out‚ 141.171875 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.drbd1716.in‚0 network.sent‚ 8.775085e+06" disk.dm-0.out‚8488.03759765625 disk.drbd2080.in‚0 disk.sda2.in‚0# memory_percent‚96.5193510055542) proc.memory.percent‚16.60282611846924 cpu‚25.341340564511242 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚8488.03759765625 disk.nvme0n1.in‚ 16.6953125( proc.memory.availableMB‚ 570.26953125# proc.memory.rssMB‚ 2720.20703125% disk./.usageGB‚235.38281631469727 disk.dm-1.in‚0* disk./.usagePercent‚15.796738302217017 disk.dm-1.out‚ 141.171875 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 16.6953125% disk.nvme0n1.out‚8488.03759765625 disk.drbd2265.in‚0 disk.sda.in‚0€´2è- _wandbruntime‚3105  _runtime‚3105å™8ç&éáM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232446296359936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257357741453312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133630929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232446278356992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3349149263872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58127946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226357647341568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3349149263872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75502981 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3513805408256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884080024892132e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58127946 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3348492888064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53083590 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336819434162 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133610863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58127948 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232446296359936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚3999 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253843770337280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75502981 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75482917 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75482917 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226357630302208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚3105.477018894 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19168922 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253843746397184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33915124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235795445623808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229694466775730 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232446278356992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33915124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235795445623808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229694466775730 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232446278356992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58127946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226357647341568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3349149263872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75502981 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3513805408256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884080024892132e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58127946 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3348492888064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53083590 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336819434162 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133610863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58127948 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232446296359936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚3999 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952Çôrœþ2ë- _wandbruntime‚3105  _runtime‚3105é˜1[ìÕ /loss/FRt1/assignment_loss‚4.083375644683838 0loss/FRt3/assignment_loss‚1.8356897842001032 2loss/SRqqt4/assignment_loss‚1.5727270566773686 1loss/FBt3/assignment_loss‚0.23771925205159317 0loss/FBt4/assignment_loss‚0.5243255563545972 0loss/SRqqt1/detection_loss‚2.378932673197526 /loss/FBt4/detection_loss‚2.3842230339845023 _step‚328 /loss/FRt1/detection_loss‚2.9970475832621255 1loss/SRqqt1/assignment_loss‚1.769483833740919 -loss/FRt4/detection_loss‚2.22479619171757 1loss/SRqqt4/detection_loss‚1.8391412889415568 /loss/FBt3/detection_loss‚0.5802260558781949 % _timestamp‚1.7884080024910483e+09 _runtime‚3105.478425428 0loss/FRt2/assignment_loss‚3.5568935982882977 2loss/SRqqt3/assignment_loss‚1.0779839337300805 0loss/FRt4/assignment_loss‚2.7110572331226788 /loss/FRt2/detection_loss‚2.1156879936655364 0loss/FBt1/assignment_loss‚0.5515363765280199 1loss/SRqqt2/detection_loss‚2.0015410078423366 %loss/total_loss‚46.17881791630609 1loss/SRqqt3/detection_loss‚1.0912087440845513 /loss/FBt2/detection_loss‚2.9949463665485383 epoch‚3 2loss/SRqqt2/assignment_loss‚1.7907540613696689 1loss/FBt2/assignment_loss‚0.43413396207615734 .loss/FRt3/detection_loss‚1.758418666543784 /loss/FBt1/detection_loss‚3.6669680178165436 trainer/global_step‚3999´E`íÚ 0loss/FRt4/assignment_loss‚2.7110572331226788 /loss/FRt2/detection_loss‚2.1156879936655364 0loss/FBt1/assignment_loss‚0.5515363765280199 1loss/SRqqt2/detection_loss‚2.0015410078423366 %loss/total_loss‚46.17881791630609 1loss/SRqqt3/detection_loss‚1.0912087440845513 /loss/FBt2/detection_loss‚2.9949463665485383 epoch‚3 2loss/SRqqt2/assignment_loss‚1.7907540613696689 1loss/FBt2/assignment_loss‚0.43413396207615734 .loss/FRt3/detection_loss‚1.758418666543784 /loss/FBt1/detection_loss‚3.6669680178165436 trainer/global_step‚3999 /loss/FRt1/assignment_loss‚4.083375644683838 0loss/FRt3/assignment_loss‚1.8356897842001032 2loss/SRqqt4/assignment_loss‚1.5727270566773686 1loss/FBt3/assignment_loss‚0.23771925205159317 0loss/FBt4/assignment_loss‚0.5243255563545972 0loss/SRqqt1/detection_loss‚2.378932673197526 /loss/FBt4/detection_loss‚2.3842230339845023 _step‚328 /loss/FRt1/detection_loss‚2.9970475832621255 1loss/SRqqt1/assignment_loss‚1.769483833740919 -loss/FRt4/detection_loss‚2.22479619171757 1loss/SRqqt4/detection_loss‚1.8391412889415568 /loss/FBt3/detection_loss‚0.5802260558781949 % _timestamp‚1.7884080024910483e+09 _runtime‚3105.478425428 0loss/FRt2/assignment_loss‚3.5568935982882977 2loss/SRqqt3/assignment_loss‚1.0779839337300805Èp–îQ’î‚(:‡ ÑáãÔ¤·‰ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µš´v[ï‚(:Ð ÑáãÔž©Ô % disk.nvme0n1.out‚8488.40087890625 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 16.6953125 disk.sda.in‚0 disk.sda.out‚ 141.7421875( proc.memory.availableMB‚ 570.26171875 network.recv‚ 1.125001e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 15.8544921875 disk.dm-0.in‚ 16.6953125 disk.sda3.in‚0$ disk./.usageGB‚235.3827247619629 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚96.51939868927002# proc.memory.rssMB‚ 2720.20703125 disk.sda2.in‚0 disk.drbd1716.in‚0" disk.dm-0.out‚8488.40087890625 disk.dm-1.out‚ 141.7421875 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 cpu‚25.339899862700523 disk.nvme0c0n1.in‚ 16.6953125' disk.nvme0c0n1.out‚8488.40087890625 disk.dm-1.in‚0 disk.sda3.out‚ 141.7421875 network.sent‚ 8.872725e+06 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0* disk./.usagePercent‚15.796732158036194 disk.sda1.out‚0) proc.memory.percent‚16.60282611846924 proc.cpu.threads‚17ô©b7’ð‚(:‡ àáãÔôÄ£ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5056 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëóUñ‚(:Ê àáãÔü¶õ $ disk./.usageGB‚235.3827247619629 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 16.6953125 disk.drbd2265.out‚0 disk.sda.out‚ 142.328125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0$ disk.nvme0n1.out‚8489.3037109375 network.recv‚ 1.130461e+06* disk./.usagePercent‚15.796732158036194 disk.dm-1.out‚ 142.328125 disk.sda.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚96.51930332183838! disk.dm-0.out‚8489.3037109375 disk.sda2.out‚0( proc.memory.availableMB‚ 570.27734375# proc.memory.rssMB‚ 2720.20703125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 142.328125& disk.nvme0c0n1.out‚8489.3037109375 disk.rbd1.in‚0 cpu‚25.338519428135225 disk.rbd1.out‚ 15.8544921875) proc.memory.percent‚16.60282611846924 network.sent‚ 8.970761e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.dm-0.in‚ 16.6953125 disk.drbd1716.in‚0{‰w2ò- _wandbruntime‚3141  _runtime‚31413:ÅAÈó ' lr-AdamW/pg2‚0.00023135782678971944 trainer/global_step‚4049 % _timestamp‚1.7884080382517352e+09 _runtime‚ 3141.23916807 _step‚329 ' lr-AdamW/pg1‚0.00023135782678971944ð€,µÍôÇ trainer/global_step‚4049 % _timestamp‚1.7884080382517352e+09 _runtime‚ 3141.23916807 _step‚329 ' lr-AdamW/pg1‚0.00023135782678971944 ' lr-AdamW/pg2‚0.00023135782678971944É€Ããm2õ- _wandbruntime‚3141  _runtime‚3141ôµ{°ö×O XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18Eß¿-'002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3388659266048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235207577260544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19390799 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238596236526592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76410553 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3556075836928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232424788077406 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376838338398 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 trainer/global_step‚4049 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3555908018688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235207595263488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389315628544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3388659266048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3376183900642 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596236526592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34315356 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238596910892032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135205245 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260420732319744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088„ðNâ'÷ÜO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34315358 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19391246 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229047949739008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3389315628544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884080382524345e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76390515 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256864656482816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3388659266048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235207577260544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19390799 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238596236526592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76410553 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3556075836928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232424788077406 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376838338398 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 trainer/global_step‚4049 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3555908018688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235207595263488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389315628544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3388659266048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3376183900642 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596236526592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34315356 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238596910892032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135205245 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260420732319744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4049 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235263930299392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260483044921344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3377654773570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58828746 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3556761640960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235263930299392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238653408993280 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238654083372032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232479837165814 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3556931543040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58828748 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76429033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19395838 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58828746 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884080389340703e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58828748 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76429033 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _step‚331 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229102853869568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229102836830208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235263948302336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53718911 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390135069696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389478693888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430Ä!¼—¶ úåM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235263948302336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53718911 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390135069696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389478693888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3377000335606 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34323526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238654083372032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235263948302336 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4049 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235263930299392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260483044921344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_trÚeOÄ5ain_batch_end/requested_bytes.small_pool.allocated‚ 3377654773570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58828746 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3556761640960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235263930299392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238653408993280 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238654083372032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232479837165814 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3556931543040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58828748 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76429033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19395838 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58828746 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884080389340703e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58828748 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76429033 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _step‚331 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229102853869568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229102836830208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ëv@6Y2û- _wandbruntime‚3141  _runtime‚3141d‰w“]ü× % _timestamp‚1.7884080389360821e+09 0loss/FRt2/assignment_loss‚3.7931032900152535 2loss/SRqqt2/assignment_loss‚1.8819517504085195 /loss/FRt4/detection_loss‚2.2273780747696206 1loss/SRqqt1/detection_loss‚2.7098671129116645 0loss/FBt3/detection_loss‚0.48745570089402784 _runtime‚3141.923449629 /loss/FRt3/detection_loss‚1.5228976211510599 0loss/SRqqt2/detection_loss‚1.695826609026302 1loss/SRqqt3/detection_loss‚1.0242595409508795 /loss/FBt1/detection_loss‚3.0130288667149014 2loss/SRqqt1/assignment_loss‚2.0174257033146343 0loss/FBt4/assignment_loss‚0.3803484042485555 /loss/FRt2/detection_loss‚2.2422367889305646 /loss/FBt4/detection_loss‚3.4795628984769187 1loss/SRqqt4/assignment_loss‚1.592234840231209 0loss/FBt1/assignment_loss‚0.5434885589276442 1loss/FBt2/assignment_loss‚0.21031106477603317 1loss/FBt3/assignment_loss‚0.17504515839531193 /loss/FRt1/detection_loss‚2.7903753672876666 1loss/SRqqt4/detection_loss‚1.7457473075081562 epoch‚3 _step‚332 /loss/FRt1/assignment_loss‚4.082921824147625 /loss/FRt4/assignment_loss‚3.432218536182686 /loss/FBt2/detection_loss‚1.8102527678012847 /loss/FRt3/assignment_loss‚1.447965151497296 2loss/SRqqt3/assignment_loss‚1.0983768071164377 %loss/total_loss‚45.40427974568425 trainer/global_step‚4049[\žbýÜ 1loss/SRqqt4/detection_loss‚1.7457473075081562 epoch‚3 _step‚332 /loss/FRt1/assignment_loss‚4.082921824147625 /loss/FRt4/assignment_loss‚3.432218536182686 /loss/FBt2/detection_loss‚1.8102527678012847 /loss/FRt3/assignment_loss‚1.447965151497296 2loss/SRqqt3/assignment_loss‚1.0983768071164377 %loss/total_loss‚45.40427974568425 trainer/global_step‚4049 % _timestamp‚1.7884080389360821e+09 0loss/FRt2/assignment_loss‚3.7931032900152535 2loss/SRqqt2/assignment_loss‚1.8819517504085195 /loss/FRt4/detection_loss‚2.2273780747696206 1loss/SRqqt1/detection_loss‚2.7098671129116645 0loss/FBt3/detection_loss‚0.48745570089402784 _runtime‚3141.923449629 /loss/FRt3/detection_loss‚1.5228976211510599 0loss/SRqqt2/detection_loss‚1.695826609026302 1loss/SRqqt3/detection_loss‚1.0242595409508795 /loss/FBt1/detection_loss‚3.0130288667149014 2loss/SRqqt1/assignment_loss‚2.0174257033146343 0loss/FBt4/assignment_loss‚0.3803484042485555 /loss/FRt2/detection_loss‚2.2422367889305646 /loss/FBt4/detection_loss‚3.4795628984769187 1loss/SRqqt4/assignment_loss‚1.592234840231209 0loss/FBt1/assignment_loss‚0.5434885589276442 1loss/FBt2/assignment_loss‚0.21031106477603317 1loss/FBt3/assignment_loss‚0.17504515839531193 /loss/FRt1/detection_loss‚2.7903753672876666Ìù¾þ‚(:… ïáãÔ˜ÍÁ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09;‚óWÿ‚(:Ì ïáãÔ೉  disk.sda3.in‚0! disk.dm-0.out‚8489.6162109375 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 16.6953125$ disk.nvme0n1.out‚8489.6162109375 disk.dm-0.in‚ 16.6953125 disk.sda.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 15.88671875 disk.sda.out‚ 143.17578125# proc.memory.rssMB‚ 2720.20703125) proc.memory.percent‚16.60282611846924 network.recv‚ 1.134685e+06* disk./.usagePercent‚15.796732158036194 disk.drbd2080.out‚0 cpu‚25.337153803262904 disk.sda3.out‚ 143.17578125 disk.sda1.in‚0 disk.drbd2265.out‚0$ disk./.usageGB‚235.3827247619629 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚96.51975631713867& proc.memory.availableMB‚ 570.203125 disk.dm-1.out‚ 143.17578125 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 network.sent‚ 8.976799e+06 disk.nvme0c0n1.in‚ 16.6953125& disk.nvme0c0n1.out‚8489.6162109375 proc.cpu.threads‚17Þ…z€‚(:’ þáãÔ«îŽ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.355' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.270999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â\EQ‚(:Æ þáãÔû¤î  disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0) proc.memory.percent‚16.60282611846924* disk./.usagePercent‚15.796723453780032& disk.nvme0c0n1.out‚8490.0947265625 disk.nvme0n1.in‚ 16.6953125 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2720.20703125 disk.sda3.in‚0( proc.memory.availableMB‚ 570.25390625$ disk.nvme0n1.out‚8490.0947265625! disk.dm-0.out‚8490.0947265625 disk.dm-1.out‚ 143.71875 disk.sda3.out‚ 143.71875 disk.sda2.out‚0 disk.rbd1.out‚ 15.98046875 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 143.71875 cpu‚25.335807017966673 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.rbd1.in‚0 disk.dm-0.in‚ 16.6953125$ memory_percent‚96.51944637298584 network.sent‚ 9.085905e+06 network.recv‚ 1.141233e+06% disk./.usageGB‚235.38259506225586 disk.drbd2266.out‚0 disk.drbd2080.out‚0–– 2‚- _wandbruntime‚3177  _runtime‚3177½ôµÇƒÁ % _timestamp‚1.7884080745043736e+09 _runtime‚3177.491963396 _step‚333 & lr-AdamW/pg1‚0.0002298965237626599 & lr-AdamW/pg2‚0.0002298965237626599 trainer/global_step‚4099ϱ2cÌ„Æ & lr-AdamW/pg1‚0.0002298965237626599 & lr-AdamW/pg2‚0.0002298965237626599 trainer/global_step‚4099 % _timestamp‚1.7884080745043736e+09 _runtime‚3177.491963396 _step‚333Ímßü2…- _wandbruntime‚3177  _runtime‚31770w}Ý'†×O UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚241454874274816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19617433 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3598737157632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136832097 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59515532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241454874274816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231793139227648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263546092050432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238025229202944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3417673677806 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241455548640256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77336605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚4099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3429645071872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238025247205888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3429645071872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54340754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238025229202944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259947169425920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238025247205888 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3430301434368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19616998 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77336605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884080745051553e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259947193366016 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414œ¿| Ï!‡ÜO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77336605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884080745051553e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259947193366016 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚241454874274816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19617433 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3598737157632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136832097 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59515532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241454874274816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231793139227648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263546092050432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238025229202944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3417673677806 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241455548640256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77336605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚4099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3429645071872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238025247205888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3429645071872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54340754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238025229202944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259947169425920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238025247205888 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3430301434368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19616998 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040Î_F‚2ˆ- _wandbruntime‚3178  _runtime‚3178¦@Uñç&‰áM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238081600244736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3431120875520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19621584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260008820163584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136864567 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884080751922524e+09 _runtime‚3178.180217403 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238081582241792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚335 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34731924 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54353951 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263608596146176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231848043358208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚241512046741504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263608393915392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235266550510546 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3418490112978 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77355085 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54353508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3417835675014 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260008844103680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59529548 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59529546 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238081600244736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235265879033222 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77335021 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3430464499712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3599573751808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34731926 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238081600244736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3431120875520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19621584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260008820163584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136864567 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884080751922524e+09 _runtime‚3178.180217403 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238081582241792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚335 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34731924 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54353951 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263608596146176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231848043358208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚241512046741504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263608393915392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235266550510546 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3418490112978 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77355085 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54353508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3417835675014 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260008844103680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59529548 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59529546 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238081600244736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235265879033222 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77335021 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3430464499712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3599573751808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34731926 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚3213.841926689 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19842382 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35132156 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234538345755648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚4149 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78262657 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3458509017214 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35132158 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78242619 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244313512023040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263029730249216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263029706309120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237996871812222 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batc)Óth_start/allocated_bytes.small_pool.freed‚ 3470630877696 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3471287240192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60216332 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60216330 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884081108543057e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240842881145344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54974969 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3471287240192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641629117952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244313512023040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234538362795008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240842899148288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244314186388480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237996200335106 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3457854579458 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266671518796800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60216332 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚338 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240842899148288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266671335427072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3470630877696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60216330 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138478989 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641788547584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54974538 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430`½\uâ'™ÜO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35132156 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234538345755648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚4149 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78262657 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3458509017214 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35132158 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78242619 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244313512023040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263029730249216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263029706309120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237996871812222 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3470630877696 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3471287240192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60216332 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60216330 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884081108543057e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240842881145344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54974969 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3471287240192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641629117952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244313512023040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234538362795008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240842899148288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244314186388480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237996200335106 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3457854579458 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266671518796800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60216332 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚338 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240842899148288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266671335427072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3470630877696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60216330 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138478989 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641788547584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54974538 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚3213.841926689 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19842382ÒÜ7ž2š- _wandbruntime‚3214  _runtime‚3214ÄÄ`–ç&›áM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138491419 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3642514506752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238052592377954 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78281137 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚339 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266734042327040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 170773504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266733871553536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4149 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240899234184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3472106681344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240899252187136 _runtime‚3214.517158837 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3471450305536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78261073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35140326 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263091357046784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60230348 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19846944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234593266925568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3459325452386 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244371358868480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244370684489728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚418 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3472106681344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54987268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60230348 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138491419 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60230346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458671014422 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3471450305536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60230346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78281137 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244371358868480 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884081115291657e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35140324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138511485 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884081115291657e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35140324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138511485 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4149 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240899234184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3472106681344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240899252187136 _runtime‚3214.517158837 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3471450305536 NADeviceStatsMonitor.on_train_batch_end/all§íkL ocation.small_pool.freed‚78261073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35140326 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263091357046784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60230348 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19846944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234593266925568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3459325452386 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244371358868480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244370684489728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚418 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3472106681344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54987268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60230348 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138491419 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60230346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458671014422 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928Ó'Ã]„2- _wandbruntime‚3214  _runtime‚3214jç“YžÓ /loss/FRt2/detection_loss‚2.1714167979856343 0loss/SRqqt2/detection_loss‚1.817239522224381 /loss/FBt2/detection_loss‚2.1877737760543825 1loss/SRqqt4/assignment_loss‚1.691496138398846 epoch‚3 trainer/global_step‚4149 0loss/FRt3/assignment_loss‚1.9053575968255803 2loss/SRqqt3/assignment_loss‚1.2269590801297454 1loss/FBt3/assignment_loss‚0.20960874796625842 /loss/SRqqt1/detection_loss‚2.26791100405358 1loss/SRqqt4/detection_loss‚1.7006014190614223 /loss/FBt1/detection_loss‚2.9908416189930658 _step‚340 0loss/FRt1/assignment_loss‚3.3424902446568012 /loss/FBt4/detection_loss‚3.4323683977127075 $ _timestamp‚1.788408111530881e+09 0loss/SRqqt3/detection_loss‚0.968801430251915 _runtime‚3214.518234387 0loss/FRt2/assignment_loss‚3.4785728628436723 0loss/FRt4/assignment_loss‚3.3025906059477066 2loss/SRqqt2/assignment_loss‚1.4815427275995414 /loss/FBt3/detection_loss‚0.6321514422799844 2loss/SRqqt1/assignment_loss‚2.0978702379239573 /loss/FRt3/detection_loss‚1.7827885824502732 0loss/FBt1/assignment_loss‚0.3275765305244766 .loss/FRt4/detection_loss‚2.311468244261212 %loss/total_loss‚45.17551588886663 0loss/FBt2/assignment_loss‚0.5556608160957694 0loss/FBt4/assignment_loss‚0.3596572964452207 /loss/FRt1/detection_loss‚2.93277076818048954ÉŒC^ŸØ 0loss/FRt4/assignment_loss‚3.3025906059477066 2loss/SRqqt2/assignment_loss‚1.4815427275995414 /loss/FBt3/detection_loss‚0.6321514422799844 2loss/SRqqt1/assignment_loss‚2.0978702379239573 /loss/FRt3/detection_loss‚1.7827885824502732 0loss/FBt1/assignment_loss‚0.3275765305244766 .loss/FRt4/detection_loss‚2.311468244261212 %loss/total_loss‚45.17551588886663 0loss/FBt2/assignment_loss‚0.5556608160957694 0loss/FBt4/assignment_loss‚0.3596572964452207 /loss/FRt1/detection_loss‚2.9327707681804895 /loss/FRt2/detection_loss‚2.1714167979856343 0loss/SRqqt2/detection_loss‚1.817239522224381 /loss/FBt2/detection_loss‚2.1877737760543825 1loss/SRqqt4/assignment_loss‚1.691496138398846 epoch‚3 trainer/global_step‚4149 0loss/FRt3/assignment_loss‚1.9053575968255803 2loss/SRqqt3/assignment_loss‚1.2269590801297454 1loss/FBt3/assignment_loss‚0.20960874796625842 /loss/SRqqt1/detection_loss‚2.26791100405358 1loss/SRqqt4/detection_loss‚1.7006014190614223 /loss/FBt1/detection_loss‚2.9908416189930658 _step‚340 0loss/FRt1/assignment_loss‚3.3424902446568012 /loss/FBt4/detection_loss‚3.4323683977127075 $ _timestamp‚1.788408111530881e+09 0loss/SRqqt3/detection_loss‚0.968801430251915 _runtime‚3214.518234387 0loss/FRt2/assignment_loss‚3.4785728628436723Ô£ Ò’ ‚(:‡ ºâãÔÝ©ö gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.818' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1636 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0róèV¡‚(:Ë ºâãÔìü  disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0) proc.memory.percent‚16.60282611846924 network.sent‚ 9.212494e+06 disk.nvme0c0n1.in‚ 16.6953125% disk.nvme0n1.out‚8492.13623046875 disk.drbd2265.in‚0$ disk./.usageGB‚235.3833999633789 disk.sda1.out‚0' disk.nvme0c0n1.out‚8492.13623046875 disk.dm-1.out‚ 146.046875 proc.cpu.threads‚17 disk.dm-0.in‚ 16.6953125 disk.sda.out‚ 146.046875( proc.memory.availableMB‚ 570.16796875 disk.sda3.out‚ 146.046875 disk.drbd2265.out‚0 disk.sda.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 16.6953125" disk.dm-0.out‚8492.13623046875$ memory_percent‚96.51997089385986# proc.memory.rssMB‚ 2720.20703125 cpu‚25.3304519522635 network.recv‚ 1.160307e+06* disk./.usagePercent‚15.796777471369754 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 16.1689453125"ìý’¢‚(:‡ ÉâãÔ–Ô gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.661' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7322 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬‰ÛW£‚(:Ì ÉâãÔÅóÒ  disk.sda3.out‚ 146.5703125 disk.sda2.in‚0 disk.sda.out‚ 146.5703125& disk.nvme0c0n1.out‚8492.7041015625 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 16.3173828125 disk.sda.in‚0# proc.memory.rssMB‚ 2720.20703125 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 16.6953125 cpu‚25.329092194721365 network.sent‚ 9.322127e+06% disk./.usageGB‚235.38336181640625 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 16.6953125 disk.dm-1.in‚0$ memory_percent‚96.51861190795898) proc.memory.percent‚16.60282611846924 disk.sda1.out‚0 network.recv‚ 1.167547e+06 disk.drbd2266.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚8492.7041015625! disk.dm-0.out‚8492.7041015625 disk.drbd2265.out‚0* disk./.usagePercent‚15.796774911294415 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 146.5703125 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-0.in‚ 16.6953125& proc.memory.availableMB‚ 570.390625 Fç2¤- _wandbruntime‚3250  _runtime‚3250Î=3[ɥà ' lr-AdamW/pg1‚0.00022691647882103945 ' lr-AdamW/pg2‚0.00022691647882103945 trainer/global_step‚4199 % _timestamp‚1.7884081471883264e+09 _runtime‚3250.175794391 _step‚341(—5Î¦È trainer/global_step‚4199 % _timestamp‚1.7884081471883264e+09 _runtime‚3250.175794391 _step‚341 ' lr-AdamW/pg1‚0.00022691647882103945 ' lr-AdamW/pg2‚0.00022691647882103945Õgzß­2§- _wandbruntime‚3250  _runtime‚3250÷ª Ü'¨ÖO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚342 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243660533087744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269796629228544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266112243192320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237283552283648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79168671 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3511616683520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79168671 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20068109 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204341248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3498689918866 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55608665 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243660551090688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884081471891046e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60917130 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140085801 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79188709 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚247172824136704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3684566437376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243660533087744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚247172149771264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269796833569792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240782242202514 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60917132 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79188709 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60917130 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55609125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3684386036224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237283569323008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266112267132416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247172149771264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3512273046016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3512273046016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35540558 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240782913679630 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243660551090688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140105841 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚247172824136704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60917132 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35540556 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096Ä™Vá'©ÛO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60917132 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79188709 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60917130 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55609125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3684386036224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237283569323008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266112267132416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247172149771264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3512273046016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3512273046016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35540558 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240782913679630 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243660551090688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140105841 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚247172824136704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60917132 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35540556 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚342 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243660533087744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269796629228544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266112243192320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237283552283648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79168671 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3511616683520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79168671 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20068109 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204341248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3498689918866 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55608665 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243660551090688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884081471891046e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60917130 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140085801 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79188709 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚247172824136704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3684566437376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243660533087744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚247172149771264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269796833569792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240782242202514 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Ö厓Ó2ª- _wandbruntime‚3250  _runtime‚3250=A_á«áM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243716904129536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3685415026688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266173893929984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79187125 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batchÒZ4 !_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3500160791794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55621325 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237338473453568 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243716886126592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35548726 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚247229322237952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60931146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35548724 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266173917870080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚4199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3250.859363474 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚343 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0oÒtì&¬æM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243716886126592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35548726 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚247229322237952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60931146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35548724 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266173917870080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚4199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3250.859363474 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚343 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243716904129536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3685415026688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266173893929984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79187125 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3500160791794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55621325 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237338473453568 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FÿDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884081833652618e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250030787519488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56243176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272922222898176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20294668 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246478185030144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250030787519488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61617930 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246478203033088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3727251064320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141712653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3540179696030 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚246478185030144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3552602489344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35948956 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269194804015616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚]M½í61617930 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35948958 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246478203033088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61617932 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56243626 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243568284069922 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61617932 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141732693 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240028775851008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80114761 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272922031139840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243568955547038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80094723 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727418882560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚4249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240028758811648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3539525258274 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20294220 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269194780075520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚3286.353053916 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3552602489344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3553258851840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872Ý«³¶â'»ÜO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141732693 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240028775851008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80114761 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272922031139840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243568955547038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80094723 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727418882560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚4249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240028758811648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3539525258274 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20294220 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269194780075520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚3286.353053916 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3552602489344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3553258851840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884081833652618e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250030787519488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56243176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272922222898176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20294668 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246478185030144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250030787519488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61617930 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246478203033088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3727251064320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141712653 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3540179696030 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚246478185030144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3552602489344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35948956 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269194804015616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61617930 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35948958 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246478203033088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61617932 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56243626 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243568284069922 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61617932 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Ú^A¼2¼- _wandbruntime‚3287  _runtime‚3287-*Pæ&½àM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240083679981568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540341693238 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250087959986176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3553421917184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3287.038467127 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246534538068992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80113177 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56255713 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246534556071936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61631946 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3554078292992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61631946 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243624676112770 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20299031 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788408184050633e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3553421917184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80133241 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3540996131202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚4249 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80133241 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61631948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243624004635446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3554078292992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35957126 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246534556071936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272984726304768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80113177 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61631948 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35957124 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¤‹Õ_ö!¾åM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243624004635446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3554078292992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35957126 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246534556071936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272984726304768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80113177 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61631948 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35957124 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240083679981568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540341693238 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250087959986176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3553421917184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3287.038467127 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246534538068992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80113177 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56255713 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246534556071936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61631946 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3554078292992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61631946 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243624676112770 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20299031 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788408184050633e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_baKpïõtch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3553421917184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80133241 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3540996131202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚4249 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80133241 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61631948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324Û*Øö2¿- _wandbruntime‚3287  _runtime‚3287_k‘5WÀÑ 1loss/FBt3/assignment_loss‚0.14071054946018005 /loss/FRt2/detection_loss‚2.2887800641003135 2loss/SRqqt1/assignment_loss‚1.6491785296372006 1loss/FBt1/assignment_loss‚0.27657453140376936 1loss/SRqqt3/detection_loss‚1.0201349298586138 0loss/FBt4/assignment_loss‚0.1790615952340886 -loss/FRt1/detection_loss‚2.77530090847323 .loss/FBt3/detection_loss‚0.543211035192186 /loss/FRt1/assignment_loss‚3.584301762042507 .loss/FRt3/assignment_loss‚1.90053375179951 %loss/total_loss‚44.69641009501258 % _timestamp‚1.7884081840525827e+09 /loss/FRt3/detection_loss‚1.7606922704726458 /loss/FRt4/detection_loss‚2.4525587348186453 1loss/SRqqt2/detection_loss‚1.9080346247484519 1loss/SRqqt4/detection_loss‚1.5649985179770738 trainer/global_step‚4249 1loss/SRqqt3/assignment_loss‚1.107853733090451 /loss/FBt1/detection_loss‚2.9701643671308244 /loss/FBt4/detection_loss‚3.0639681965112686 epoch‚3 _step‚348 0loss/FRt2/assignment_loss‚3.7009245668138777 /loss/FRt4/assignment_loss‚3.547212348036144 1loss/SRqqt2/assignment_loss‚1.454566433679226 1loss/FBt2/assignment_loss‚0.31494706742814743 0loss/SRqqt1/detection_loss‚2.242445181097303 _runtime‚3287.039973961 1loss/SRqqt4/assignment_loss‚1.883427839151409 /loss/FBt2/detection_loss‚2.3668285568555194dÆæ2\ÁÖ 1loss/SRqqt3/detection_loss‚1.0201349298586138 0loss/FBt4/assignment_loss‚0.1790615952340886 -loss/FRt1/detection_loss‚2.77530090847323 .loss/FBt3/detection_loss‚0.543211035192186 /loss/FRt1/assignment_loss‚3.584301762042507 .loss/FRt3/assignment_loss‚1.90053375179951 %loss/total_loss‚44.69641009501258 % _timestamp‚1.7884081840525827e+09 /loss/FRt3/detection_loss‚1.7606922704726458 /loss/FRt4/detection_loss‚2.4525587348186453 1loss/SRqqt2/detection_loss‚1.9080346247484519 1loss/SRqqt4/detection_loss‚1.5649985179770738 trainer/global_step‚4249 1loss/SRqqt3/assignment_loss‚1.107853733090451 /loss/FBt1/detection_loss‚2.9701643671308244 /loss/FBt4/detection_loss‚3.0639681965112686 epoch‚3 _step‚348 0loss/FRt2/assignment_loss‚3.7009245668138777 /loss/FRt4/assignment_loss‚3.547212348036144 1loss/SRqqt2/assignment_loss‚1.454566433679226 1loss/FBt2/assignment_loss‚0.31494706742814743 0loss/SRqqt1/detection_loss‚2.242445181097303 _runtime‚3287.039973961 1loss/SRqqt4/assignment_loss‚1.883427839151409 /loss/FBt2/detection_loss‚2.3668285568555194 1loss/FBt3/assignment_loss‚0.14071054946018005 /loss/FRt2/detection_loss‚2.2887800641003135 2loss/SRqqt1/assignment_loss‚1.6491785296372006 1loss/FBt1/assignment_loss‚0.27657453140376936Ü ¿Á»’‚(:‡ …ããÔ¦•– gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.033' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6066 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\6ѯQÂ(:Æ …ããÔ¦¡ã  network.sent‚ 9.449228e+06 network.recv‚ 1.186534e+06! disk.dm-0.out‚8494.7802734375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚8494.7802734375% disk./.usageGB‚235.38275527954102 disk.rbd1.in‚0 disk.rbd1.out‚ 16.6220703125 disk.dm-0.in‚ 16.6953125 disk.drbd2266.out‚0 disk.rbd0.in‚0$ memory_percent‚96.52466773986816' proc.memory.availableMB‚ 569.3984375* disk./.usagePercent‚15.796734206096469 disk.nvme0n1.in‚ 16.6953125$ disk.nvme0n1.out‚8494.7802734375 disk.sda3.out‚ 148.96875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 148.96875 disk.drbd1716.in‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2720.20703125) proc.memory.percent‚16.60282611846924 cpu‚25.32397874238931 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.sda.out‚ 148.96875 disk.dm-1.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 16.6953125 proc.cpu.threads‚17¸ùÞ’Ä‚(:‡ ”ããÔ¦¨ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž_¨”[Å‚(:Ð ”ããÔŽ – # proc.memory.rssMB‚ 2720.20703125 disk.dm-1.out‚ 149.33203125 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 16.6953125$ memory_percent‚96.52318954467773 network.recv‚ 1.192786e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 16.6953125) proc.memory.percent‚16.60282611846924 disk.drbd2080.out‚0 disk.rbd1.out‚ 16.6220703125 disk.sda.out‚ 149.33203125 cpu‚25.32274197274855 network.sent‚ 9.55772e+06 disk.sda3.out‚ 149.33203125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚8495.19189453125 disk.drbd2265.in‚0& proc.memory.availableMB‚ 569.640625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚8495.19189453125" disk.dm-0.out‚8495.19189453125 disk.sda.in‚0* disk./.usagePercent‚15.796734462104004% disk./.usageGB‚235.38275909423828 disk.dm-0.in‚ 16.6953125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚00™iÓ2Æ- _wandbruntime‚3322  _runtime‚3322¬ GÁÇÇÁ % _timestamp‚1.7884082199707565e+09 _runtime‚3322.958206691 _step‚349 & lr-AdamW/pg1‚0.0002238619253326126 & lr-AdamW/pg2‚0.0002238619253326126 trainer/global_step‚4299¯Î¦ÌÈÆ & lr-AdamW/pg1‚0.0002238619253326126 & lr-AdamW/pg2‚0.0002238619253326126 trainer/global_step‚4299 % _timestamp‚1.7884082199707565e+09 _runtime‚3322.958206691 _step‚349ÝÕ5äh2É- _wandbruntime‚3322  _runtime‚3322²ë€"Ý'Ê×O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20519501 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62318730 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20519952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚3322.959125195 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56877310 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 trainer/global_step‚4299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62318732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143359545 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81040813 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242773982379008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242773965339648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3581015035438 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3594244657664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249295854975488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252890099633152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276047546177024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252889425267712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3770397036544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276047737935360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143339505 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62318730 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246354997414446ždö:â'ËÜO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81020775 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249295836972544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚350 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249295854975488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3594244657664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3593588295168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81040813 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62318732 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252889425267712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3770397036544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276047737935360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143339505 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62318730 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246354997414446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20519501 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62318730 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20519952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚3322.959125195 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56877310 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 trainer/global_step‚4299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62318732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143359545 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81040813 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242773982379008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242773965339648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3581015035438 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3594244657664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249295854975488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252890099633152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276047546177024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81039229 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81059293 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252947272113152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62332748 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36365526 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249352208014336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3594407723008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56889568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272338991636480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20524044 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771249656320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3581177032646 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3771079754šÇfç240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3323.648383385 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143371975 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3594407723008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246410046502854 _step‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62332748 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3595064098816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81059293 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3581831470610 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56890015 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62332746 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249352190011392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3595064098816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252946597734400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844+9÷ê&ÎäM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249352190011392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81039229 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81059293 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252947272113152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62332748 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36365526 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249352208014336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3594407723008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56889568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272338991636480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20524044 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771249656320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3581177032646 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3771079754240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3323.648383385 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143371975 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3594407723008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246410046502854 _step‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62332748 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3595064098816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81059293 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3581831470610 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56890015 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62332746 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249352190011392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3595064098816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252946597734400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252113506917888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3634574100992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63019530 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20745906 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20745463 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279172972450816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252113488914944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279173162112000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255748063015936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3635230463488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252113488914944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57511664 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634574100992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3813118608896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144966357 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81946827 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252113506917888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249141039281854 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3621850374846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048þf"1ÛÚO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63019532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255748063015936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144986397 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252113506917888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3634574100992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceSta¬•¯$tsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63019530 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20745906 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20745463 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279172972450816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252113488914944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279173162112000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255748063015936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3635230463488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252113488914944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57511664 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634574100992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3813118608896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144966357 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81946827 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252113506917888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249141039281854 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3621850374846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 # _timestamp‚1.78840825626759e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63019532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4349 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255748737381376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63019530 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚3359.255495142 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255748737381376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245519171867648â["ö¬2Ü- _wandbruntime‚3359  _runtime‚3359¿Aãæ&ÝàM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3814141783040 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57523830 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255805909861376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4349 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63033546 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81985345 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3635393528832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249196759847586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57524276 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145018893 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635393528832 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3622666810018 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3636049904640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36773924 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3622012372054 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255805909861376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788408256945568e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63033548 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255805235482624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252169841953792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81965281 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚355 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36773926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3359.933449954 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252169859956736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20749906 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275421528519680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144998827 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279235670302720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252169841953792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3636049904640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63033546 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245574093037568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252169859956736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275421504579584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279235480654848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3813976075264 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81965281 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279235480654848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3813976075264 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81965281 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57523830 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255805909861376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4349 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63033546 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81985345 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3635393528832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249196759847586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57524276 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145018893 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635393528832 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3622666810018 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3636049904640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36773924 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3622012372054 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255805909861376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788408256945568e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63033548 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255805235482624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252169841953792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81965281 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚355 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36773926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3359.933449954 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252169859956736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20749906 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275421528519680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144998827 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279235670302720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928ã/Uoæ2ß- _wandbruntime‚3359  _runtime‚3359Šƒ­XXàÒ /loss/FRt2/detection_loss‚2.0893703280612543 /loss/FRt3/detection_loss‚1.7064121888577939 /loss/FBt2/detection_loss‚2.8381648626592426 2loss/SRqqt1/assignment_loss‚1.9454771298915148 /loss/FBt1/assignment_loss‚0.391263113868384 1loss/SRqqt1/detection_loss‚2.4642448984086514 1loss/SRqqt2/assignment_loss‚1.642338762210833 2loss/SRqqt3/assignment_loss‚1.1278349369250495 .loss/FRt4/detection_loss‚2.296306947246194 %loss/total_loss‚44.15596319432223 % _timestamp‚1.7884082569471388e+09 0loss/FRt3/assignment_loss‚1.7385084700584412 1loss/FBt4/assignment_loss‚0.21684098052875406 /loss/FRt1/detection_loss‚3.1522763812983476 1loss/SRqqt3/detection_loss‚0.7605120460663376 /loss/FRt1/assignment_loss‚4.057627006813332 0loss/FRt4/assignment_loss‚2.4687793788810573 2loss/SRqqt4/assignment_loss‚1.4656514087826877 1loss/FBt2/assignment_loss‚0.24238400533795357 /loss/FBt1/detection_loss‚2.8100099484125773 .loss/FBt4/detection_loss‚2.364623103822981 _step‚356 /loss/FRt2/assignment_loss‚4.066389761072524 0loss/SRqqt4/detection_loss‚1.507318490009377 epoch‚3 trainer/global_step‚4349 _runtime‚3359.934803459 /loss/FBt3/assignment_loss‚0.214626689503048 0loss/SRqqt2/detection_loss‚2.005662122288266 /loss/FBt3/detection_loss‚0.5833402333176299ä?QY]á× /loss/FRt1/assignment_loss‚4.057627006813332 0loss/FRt4/assignment_loss‚2.4687793788810573 2loss/SRqqt4/assignment_loss‚1.4656514087826877 1loss/FBt2/assignment_loss‚0.24238400533795357 /loss/FBt1/detection_loss‚2.8100099484125773 .loss/FBt4/detection_loss‚2.364623103822981 _step‚356 /loss/FRt2/assignment_loss‚4.066389761072524 0loss/SRqqt4/detection_loss‚1.507318490009377 epoch‚3 trainer/global_step‚4349 _runtime‚3359.934803459 /loss/FBt3/assignment_loss‚0.214626689503048 0loss/SRqqt2/detection_loss‚2.005662122288266 /loss/FBt3/detection_loss‚0.5833402333176299 /loss/FRt2/detection_loss‚2.0893703280612543 /loss/FRt3/detection_loss‚1.7064121888577939 /loss/FBt2/detection_loss‚2.8381648626592426 2loss/SRqqt1/assignment_loss‚1.9454771298915148 /loss/FBt1/assignment_loss‚0.391263113868384 1loss/SRqqt1/detection_loss‚2.4642448984086514 1loss/SRqqt2/assignment_loss‚1.642338762210833 2loss/SRqqt3/assignment_loss‚1.1278349369250495 .loss/FRt4/detection_loss‚2.296306947246194 %loss/total_loss‚44.15596319432223 % _timestamp‚1.7884082569471388e+09 0loss/FRt3/assignment_loss‚1.7385084700584412 1loss/FBt4/assignment_loss‚0.21684098052875406 /loss/FRt1/detection_loss‚3.1522763812983476 1loss/SRqqt3/detection_loss‚0.7605120460663376ä«|¯ç‘â‚(:† ÁããÔÆÝ’ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.373 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ˜ÍŠ—ã‚(:Ê ÁããÔ’§Ú  disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 cpu‚25.319049708977467 disk.dm-1.out‚ 154.63671875¸c­¾ disk.sda1.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 569.546875) proc.memory.percent‚16.60282611846924$ disk.nvme0n1.out‚8504.9462890625 disk.nvme0c0n1.in‚ 16.6953125& disk.nvme0c0n1.out‚8504.9462890625 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚0! disk.dm-0.out‚8504.9462890625* disk./.usagePercent‚15.796948740410164$ disk./.usageGB‚235.3859519958496 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0$ memory_percent‚96.52376174926758 disk.rbd1.out‚ 16.810546875 network.sent‚ 9.678e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 16.6953125 proc.cpu.threads‚17 network.recv‚ 1.207087e+06# proc.memory.rssMB‚ 2720.20703125 disk.drbd2266.out‚0 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 154.63671875 disk.sda3.out‚ 154.63671875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 16.6953125£}œä‚(:‘ ÐããÔÂ…µ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.569' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.71379999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò¢ÓSå‚(:È ÐããԺɚ  disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda2.in‚0 network.sent‚ 9.786613e+06 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 16.896484375 network.recv‚ 1.213195e+06" disk.nvme0n1.out‚ 8508.23046875 disk.dm-0.out‚ 8508.23046875 disk.dm-1.out‚ 155.10546875 disk.sda3.out‚ 155.10546875 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.dm-0.in‚ 16.6953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚15.796950020447834% disk./.usageGB‚235.38597106933594 disk.sda.out‚ 155.10546875 disk.drbd2266.in‚0 disk.rbd0.in‚0$ memory_percent‚96.52390480041504' proc.memory.availableMB‚ 569.5234375# proc.memory.rssMB‚ 2720.20703125 disk.nvme0n1.in‚ 16.6953125) proc.memory.percent‚16.60282611846924 cpu‚25.31785488214474 disk.drbd2265.in‚0$ disk.nvme0c0n1.out‚ 8508.23046875»±W’æ‚(:‡ ßããÔô‘¬ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒMïšUç‚(:Ê ßããÔÓßÙ  disk.rbd1.in‚0 disk.sda3.out‚ 155.7109375 disk.nvme0c0n1.in‚ 16.6953125& disk.nvme0c0n1.out‚8508.8369140625$ memory_percent‚96.52543067932129# proc.memory.rssMB‚ 2720.20703125 disk.sda1.in‚0) disk./.usagePercent‚15.79694438828208 disk.dm-0.in‚ 16.6953125 disk.sda.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 16.958984375 disk.nvme0n1.in‚ 16.6953125! disk.dm-0.out‚8508.8369140625 disk.drbd2265.in‚0 disk.sda.out‚ 155.7109375$ disk./.usageGB‚235.3858871459961 disk.drbd2265.out‚0 disk.drbd2266.in‚0) proc.memory.percent‚16.60282611846924 disk.rbd0.out‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚8508.8369140625 disk.dm-1.out‚ 155.7109375 disk.rbd0.in‚0 network.sent‚ 9.792575e+06 network.recv‚ 1.217492e+06 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0' proc.memory.availableMB‚ 569.2734375 cpu‚25.316686254323198ÅŠÏ2è- _wandbruntime‚3395  _runtime‚3395 dÕÛÉéà ' lr-AdamW/pg1‚0.00022073540942219507 ' lr-AdamW/pg2‚0.00022073540942219507 trainer/global_step‚4399 % _timestamp‚1.7884082927897658e+09 _runtime‚3395.777214738 _step‚357ØÂÞ·ÎêÈ ' lr-AdamW/pg1‚0.00022073540942219507 ' lr-AdamW/pg2‚0.00022073540942219507 trainer/global_step‚4399 % _timestamp‚1.7884082927897658e+09 _runtime‚3395.777214738 _step‚357åg²…2ë- _wandbruntime‚3395  _runtime‚3395…ªÜ'ìÖO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63720330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258606700764160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37174158 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282298441264128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3856050539008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58146609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248264378395648 trainer/global_step‚4399 $ _timestamp‚1.788408292790484e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82892917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278442390725120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3676216269312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82872879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3856222551552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82872879 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37174156 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3662031276498 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3676216269312 _step‚358 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63720332 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251926409672146 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258607375129600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚282298637216768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248264395435008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3675559906816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚254931140857344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82892917 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚3395.778118884 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚258607375129600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254931158860288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146593209 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63720332 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20972894 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278442414665216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254931140857344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3675559906816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20972453 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251927081149262 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚254931158860288’ŒDá'íÛO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82872879 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37174156 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3662031276498 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3676216269312 _step‚358 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63720332 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251926409672146 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258607375129600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚282298637216768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248264395435008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3675559906816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚254931140857344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82892917 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚3395.778118884 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚258607375129600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254931158860288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146593209 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63720332 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20972894 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278442414665216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254931140857344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3675559906816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20972453 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251927081149262 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚254931158860288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63720330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258606700764160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37174158 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282298441264128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3856050539008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58146609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248264378395648 trainer/global_step‚4399 $ _timestamp‚1.788408292790484e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82892917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278442390725120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3676216269312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82872879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3856222551552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0æ‹7l2î- _wandbruntime‚3396  _runtime‚3396žÆù´'ïàM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37182326 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251982801714994 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3663502149426 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚258664547609600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚4399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63734348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278504065402880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚248319282526208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788408293472592e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3857081773056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚258663873230848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37182324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3677035710464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159303 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63734346 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82911397 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3662847711462 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3856926551040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82891333 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146625679 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63734348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278504065402880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚248319282526208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788408293472592e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3857081773056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚258663873230848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37182324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3677035710464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159303 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63734346 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82911397 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3662847711462 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3856926551040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82891333 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146625679 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37182326 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251982801714994 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3663502149426 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚258664547609600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚4399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148240101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261466012877824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261466012877824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37582558 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251009584923648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257748810802688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3716545712640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261465338512384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83798931 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261465338512384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251009601963008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257748792799744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3899262718464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254713123016670 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21200054 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3702866615906 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚4449 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884083287389617e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64421132 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3703521053662 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148220061 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58782612 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3717202075136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64421130 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285424214266880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _runtime‚3431.726606738 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3716545712640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21200054 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3702866615906 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚4449 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884083287389617e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64421132 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3703521053662 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148220061 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58782612 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3717202075136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64421130 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285424214266880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _runtime‚3431.726606738 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3716545712640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148240101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261466012877824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261466012877824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37582558 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251009584923648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257748810802688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3716545712640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261465338512384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2êò hæ2þ- _wandbruntime‚3432  _runtime‚3432’©ç&ÿáM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285486724358144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚363 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3717365140480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281586602286080 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3900122072064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚3432.416048703 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64435148 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64435146 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83837449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21204120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64435148 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281586578345984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3703683050870 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261523185357824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884083294278746e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261522510979072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3899960558592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37590724 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251064489054208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚285486538904576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58794844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148252531 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257805163841536 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3717365140480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37590726 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83837449 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257805145838592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261523185357824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83817385 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83817385 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251064506093568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254768843582402 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254768172105078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3718021516288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21204546 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3717365140480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37590726 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83837449 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257805145838592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261523185357824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83817385 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83817385 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251064506093568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254768843582402 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254768172105078 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3718021516288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21204546 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚363 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3717365140480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281586602286080 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3900122072064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚3432.416048703 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64435148 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64435146 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83837449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21204120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64435148 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281586578345984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3703683050870 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261523185357824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884083294278746e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261522510979072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3899960558592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37590724 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251064489054208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚285486538904576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58794844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148252531 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257805163841536 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264323976260608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264324650626048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84724983 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253754808491008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21425220 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284607464491520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3942063425536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚4499 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264323976260608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758187880960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260566444742144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59416624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21425666 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3757531518464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253754791451648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288549721772544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284607488431616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788408365201056e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59416176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288549527917056 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260566462745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65121930 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757531518464 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚366 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3942233340928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84724983 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84745021 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65121932 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149846913 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84745021÷à™@á'ÛO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288549527917056 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260566462745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65121930 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757531518464 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚366 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3942233340928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84724983 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84745021 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65121932 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149846913 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84745021 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264324650626048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚3468.188675618 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3744356393070 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3758187880960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚257499164884078 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260566444742144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3743701955314 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264323976260608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264324650626048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84724983 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253754808491008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21425220 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284607464491520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3942063425536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚4499 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264323976260608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758187880960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260566444742144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59416624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21425666 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3757531518464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253754791451648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288549721772544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284607488431616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788408365201056e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59416176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0î¬E™62- _wandbruntime‚3468  _runtime‚3468l™ì5ç&‘áM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59428804 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149879383 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37999124 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65135946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21429680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260622797780992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚4499 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3759007322112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚253809695582208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3744518390278 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚3468.866683576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758350946304 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257554885449810 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260622815783936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚264381823106048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚264381148727296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149899449 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884083658788037e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759007322112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288612226225664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3943087056384 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84743437 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3745172828242 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879383 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84743437 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288612036577792 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149899449 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884083658788037e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759007322112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288612226225664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3943087056384 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84743437 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3745172828242 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879383 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84743437 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288612036577792 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59428804 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149879383 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37999124 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65135946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21429680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260622797780992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚4499 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3759007322112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚253809695582208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3744518390278 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚3468.866683576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758350946304 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257554885449810 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872ïØ2|2“- _wandbruntime‚3468  _runtime‚3468±½=V”Ð /loss/FRt1/detection_loss‚3.0042448500792185 0loss/SRqqt1/detection_loss‚2.332157878412141 .loss/FBt2/detection_loss‚2.880052152276039 %loss/total_loss‚43.40888504370865 .loss/FBt3/detection_loss‚0.569437143202363 trainer/global_step‚4499 _runtime‚3468.867727736 1loss/SRqqt1/assignment_loss‚1.874929244733519 0loss/FBt1/assignment_loss‚0.5235333236159931 1loss/SRqqt4/detection_loss‚1.6241817025002092 _step‚368 1loss/SRqqt3/detection_loss‚1.2391101371031255 -loss/FBt1/detection_loss‚2.41817370057106 .loss/FBt4/detection_loss‚2.371398001909256 2loss/SRqqt2/assignment_loss‚1.6125462177269003 2loss/SRqqt3/assignment_loss‚1.1483079829195049 2loss/SRqqt4/assignment_loss‚1.6680472540901974 1loss/FBt3/assignment_loss‚0.15888383442209839 .loss/FRt2/detection_loss‚2.227695660157637 /loss/FRt4/detection_loss‚2.6114728392624276 /loss/FRt3/assignment_loss‚1.532344072448964 epoch‚3 0loss/FRt2/assignment_loss‚2.9193143411116167 1loss/FBt2/assignment_loss‚0.24738111896440315 1loss/FBt4/assignment_loss‚0.19153665545309195 .loss/FRt3/detection_loss‚1.610402837395668 $ _timestamp‚1.788408365880367e+09 0loss/FRt1/assignment_loss‚3.3993296066919965 /loss/FRt4/assignment_loss‚3.467380468438311 1loss/SRqqt2/detection_loss‚1.7770240202229073îëç[•Õ 0loss/FRt1/assignment_loss‚3.3993296066919965 /loss/FRt4/assignment_loss‚3.467380468438311 1loss/SRqqt2/detection_loss‚1.7770240202229073 /loss/FRt1/detection_loss‚3.0042448500792185 0loss/SRqqt1/detection_loss‚2.332157878412141 .loss/FBt2/detection_loss‚2.880052152276039 %loss/total_loss‚43.40888504370865 .loss/FBt3/detection_loss‚0.569437143202363 trainer/global_step‚4499 _runtime‚3468.867727736 1loss/SRqqt1/assignment_loss‚1.874929244733519 0loss/FBt1/assignment_loss‚0.5235333236159931 1loss/SRqqt4/detection_loss‚1.6241817025002092 _step‚368 1loss/SRqqt3/detection_loss‚1.2391101371031255 -loss/FBt1/detection_loss‚2.41817370057106 .loss/FBt4/detection_loss‚2.371398001909256 2loss/SRqqt2/assignment_loss‚1.6125462177269003 2loss/SRqqt3/assignment_loss‚1.1483079829195049 2loss/SRqqt4/assignment_loss‚1.6680472540901974 1loss/FBt3/assignment_loss‚0.15888383442209839 .loss/FRt2/detection_loss‚2.227695660157637 /loss/FRt4/detection_loss‚2.6114728392624276 /loss/FRt3/assignment_loss‚1.532344072448964 epoch‚3 0loss/FRt2/assignment_loss‚2.9193143411116167 1loss/FBt2/assignment_loss‚0.24738111896440315 1loss/FBt4/assignment_loss‚0.19153665545309195 .loss/FRt3/detection_loss‚1.610402837395668 $ _timestamp‚1.788408365880367e+09ð˜UH¸œ–‚(:‘ ¹äãÔžßø gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.89' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.777999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œNY—‚(:Î ¹äãÔïë„  disk.sda.in‚0 disk.sda.out‚ 159.39453125 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0$ disk./.usageGB‚235.3863754272461 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0% disk.nvme0c0n1.out‚8511.537109375 network.sent‚ 1.0034763e+07 disk.sda3.in‚0# proc.memory.rssMB‚ 2720.20703125) proc.memory.percent‚16.60282611846924 disk.dm-0.out‚8511.537109375 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 17.3896484375# disk.nvme0n1.out‚8511.537109375 disk.dm-0.in‚ 16.6953125 disk.dm-1.in‚0 disk.sda3.out‚ 159.39453125 disk.sda1.in‚0$ memory_percent‚96.52612209320068 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 16.6953125( proc.memory.availableMB‚ 569.16015625 proc.cpu.threads‚17* disk./.usagePercent‚15.796977157246465 disk.nvme0n1.in‚ 16.6953125 disk.dm-1.out‚ 159.39453125 cpu‚25.309690668539485 network.recv‚ 1.246979e+06 disk.drbd2265.in‚0¦ð Ì’˜‚(:‡ ÈäãÔ¾‘Œ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚134.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßçn\™‚(:Ñ ÈäãÔÌÕ¬  disk.sda.out‚ 162.30078125 disk.sda3.in‚0) proc.memory.percent‚16.60282611846924 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 16.6953125 disk.drbd2265.out‚0 disk.sda3.out‚ 162.30078125 disk.drbd2080.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚8514.7548828125 disk.dm-1.in‚0 disk.drbd2265.in‚0 network.sent‚ 1.0143109e+07 network.recv‚ 1.252745e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0& disk.nvme0c0n1.out‚8514.7548828125 disk.drbd1716.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚8514.7548828125$ memory_percent‚96.52478694915771( proc.memory.availableMB‚ 569.37890625# proc.memory.rssMB‚ 2720.20703125 cpu‚25.30852340795103 disk.rbd0.out‚0 disk.dm-1.out‚ 162.30078125* disk./.usagePercent‚15.618911884828233 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 16.6953125 proc.cpu.threads‚17% disk./.usageGB‚232.73307418823242 disk.nvme0c0n1.in‚ 16.6953125 disk.rbd1.out‚ 17.3896484375¢­²>2š- _wandbruntime‚3504  _runtime‚3504h#ɛà ' lr-AdamW/pg1‚0.00021591642274743016 ' lr-AdamW/pg2‚0.00021591642274743016 trainer/global_step‚4549 % _timestamp‚1.7884084013592844e+09 _runtime‚3504.346881442 _step‚369æ!ÂSÎœÈ ' lr-AdamW/pg2‚0.00021591642274743016 trainer/global_step‚4549 % _timestamp‚1.7884084013592844e+09 _runtime‚3504.346881442 _step‚369 ' lr-AdamW/pg1‚0.00021591642274743016ñYlØ$2- _wandbruntime‚3504  _runtime‚3504À`ŠÝ'ž×O N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151493805 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85651035 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3799173686784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60050256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38399358 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3984929799680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287690001374720 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260284535274370 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚4549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884084013600843e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151473765 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267183288374272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291674931174400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287690025314816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267183288374272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263384114687488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267182614008832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256500015019008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38399356 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ÒUž`CŸÜO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151473765 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267183288374272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291674931174400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287690025314816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267183288374272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263384114687488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267182614008832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256500015019008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38399356 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151493805 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85651035 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batc¿„±Ÿ h_start/allocated_bytes.small_pool.allocated‚ 3799173686784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60050256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38399358 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3984929799680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287690001374720 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260284535274370 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚4549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884084013600843e+09òÍñE:2 - _wandbruntime‚3505  _runtime‚3505m=”ç&¡áM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256554919149568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263440467726336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3799336752128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3799993127936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85669489 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291737620271104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267239786475520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65836748 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60062865 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260340255839894 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287751652112384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256554902110208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3786008167650 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65836748 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151506235 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85689553 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85669489 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151526301 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263440449723392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚371 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65836746 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267240460854272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151506235 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291737424331776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884084020448084e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38407524 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260340927317218 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60062416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21654892 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267240460854272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263440449723392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85689553 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3985772219392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287751676052480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3785353729686 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263440467726336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2uX«ì&¢æM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287751676052480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3785353729686 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263440467726336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256554919149568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263440467726336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3799336752128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3799993127936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85669489 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291737620271104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267239786475520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65836748 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60062865 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260340255839894 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287751652112384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256554902110208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3786008167650 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65836748 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151506235 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85689553 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85669489 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151526301 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263440449723392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚371 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65836746 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267240460854272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151506235 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291737424331776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884084020448084e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38407524 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260340927317218 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60062416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21654892 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267240460854272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263440449723392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85689553 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3985772219392ó¹†Üp2£- _wandbruntime‚3505  _runtime‚3505ls82Z¤Ô 2loss/SRqqt4/assignment_loss‚1.6394666920402157 /loss/FRt1/detection_loss‚3.1508324795535634 .loss/FRt3/detection_loss‚1.833675698786974 /loss/FRt4/detection_loss‚2.2163203274994565 trainer/global_step‚4549 /loss/FRt3/assignment_loss‚1.505036016702652 2loss/FBt4/assignment_loss‚0.040511394302614336 0loss/SRqqt4/detection_loss‚1.571451783847453 %loss/total_loss‚45.19770600063538 /loss/FBt2/detection_loss‚1.9837137865168708 _step‚372 0loss/FRt2/assignment_loss‚2.9048845881507512 $ _timestamp‚1.788408402046415e+09 epoch‚3 2loss/SRqqt3/assignment_loss‚1.1198403919270883 /loss/FRt2/detection_loss‚2.3641534518627894 1loss/SRqqt2/detection_loss‚1.8878676721027918 /loss/FBt1/detection_loss‚2.2270461045778713 0loss/FRt4/assignment_loss‚3.0425009225544177 2loss/SRqqt1/assignment_loss‚2.1096085617619176 2loss/SRqqt2/assignment_loss‚1.6918957611945058 1loss/SRqqt1/detection_loss‚2.7480428238068857 _runtime‚3505.033796058 1loss/FBt1/assignment_loss‚0.38970391678762656 0loss/FBt2/assignment_loss‚0.7223042288845006 /loss/FBt3/detection_loss‚0.7021052197921147 -loss/FRt1/assignment_loss‚4.2871424640928 0loss/FBt3/assignment_loss‚0.2306781441368181 1loss/SRqqt3/detection_loss‚1.2508041044697165 .loss/FBt4/detection_loss‚3.578119465282985ßö»ß_¥Ù -loss/FRt1/assignment_loss‚4.2871424640928 0loss/FBt3/assignment_loss‚0.2306781441368181 1loss/SRqqt3/detection_loss‚1.2508041044697165 .loss/FBt4/detection_loss‚3.578119465282985 2loss/SRqqt4/assignment_loss‚1.6394666920402157 /loss/FRt1/detection_loss‚3.1508324795535634 .loss/FRt3/detection_loss‚1.833675698786974 /loss/FRt4/detection_loss‚2.2163203274994565 trainer/global_step‚4549 /loss/FRt3/assignment_loss‚1.505036016702652 2loss/FBt4/assignment_loss‚0.040511394302614336 0loss/SRqqt4/detection_loss‚1.571451783847453 %loss/total_loss‚45.19770600063538 /loss/FBt2/detection_loss‚1.9837137865168708 _step‚372 0loss/FRt2/assignment_loss‚2.9048845881507512 $ _timestamp‚1.788408402046415e+09 epoch‚3 2loss/SRqqt3/assignment_loss‚1.1198403919270883 /loss/FRt2/detection_loss‚2.3641534518627894 1loss/SRqqt2/detection_loss‚1.8878676721027918 /loss/FBt1/detection_loss‚2.2270461045778713 0loss/FRt4/assignment_loss‚3.0425009225544177 2loss/SRqqt1/assignment_loss‚2.1096085617619176 2loss/SRqqt2/assignment_loss‚1.6918957611945058 1loss/SRqqt1/detection_loss‚2.7480428238068857 _runtime‚3505.033796058 1loss/FBt1/assignment_loss‚0.38970391678762656 0loss/FBt2/assignment_loss‚0.7223042288845006 /loss/FBt3/detection_loss‚0.7021052197921147ôÖ·C•’¦‚(:‡ ×äãԺĈ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01áhL§‚(:Á ×äãÔ‰ŠÂ  disk.rbd0.in‚0# proc.memory.rssMB‚ 2720.20703125 disk.nvme0n1.in‚ 16.6953125 disk.drbd2265.out‚0( proc.memory.availableMB‚ 569.13671875 proc.cpu.threads‚17 disk.drbd2080.in‚0 network.sent‚ 1.014905e+07* disk./.usagePercent‚15.618943117747408 disk.drbd1716.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚164.3125$ memory_percent‚96.52626514434814 network.recv‚ 1.256934e+06 disk.rbd1.out‚ 17.421875$ disk.nvme0n1.out‚8520.1748046875 disk.sda.in‚0 disk.sda2.in‚0) proc.memory.percent‚16.60282611846924& disk.nvme0c0n1.out‚8520.1748046875 disk.rbd1.in‚0 disk.dm-1.out‚164.3125 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd1716.out‚0 disk.sda.out‚164.3125 disk.drbd2266.out‚0 cpu‚25.307423952337867% disk./.usageGB‚232.73353958129883 disk.dm-0.in‚ 16.6953125 disk.sda1.in‚0! disk.dm-0.out‚8520.1748046875 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0m¨y¨‚(:’ æäãÔÚò– gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.328' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.665600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BÉ#oZ©‚(:Ï æäãÔðƒ™  disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚8525.49072265625 disk.dm-1.out‚ 164.8828125 disk.drbd2265.in‚0% disk./.usageGB‚232.73335647583008 disk.sda3.out‚ 164.8828125 disk.sda1.out‚0 network.sent‚ 1.0258227e+07 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 16.6953125" disk.dm-0.out‚8525.49072265625 network.recv‚ 1.263581e+06 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 16.6953125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 164.8828125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚8525.49072265625 disk.drbd2265.out‚0$ memory_percent‚96.52349948883057( proc.memory.availableMB‚ 569.58984375 cpu‚25.306311403003036* disk./.usagePercent‚15.618930829385766 disk.rbd0.in‚0 disk.nvme0n1.in‚ 16.6953125# proc.memory.rssMB‚ 2720.20703125) proc.memory.percent‚16.60282611846924 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 17.6015625.“t’ª‚(:‡ õäãÔ‡½— gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.776' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“e?sI«‚(:¾ õäãÔ³ª˜ $ disk.nvme0c0n1.out‚ 8525.83984375 disk.rbd1.out‚17.75 disk.dm-0.in‚ 16.6953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0* disk./.usagePercent‚15.618934925506315 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 16.6953125 disk.drbd1716.in‚0 network.sent‚ 1.0264487e+07 disk.dm-1.in‚0 disk.drbd2080.in‚0) proc.memory.percent‚16.60282611846924 cpu‚25.305233060653148 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚96.52507305145264# proc.memory.rssMB‚ 2720.20703125% disk./.usageGB‚232.73341751098633 disk.sda2.out‚0 disk.drbd1716.out‚0" disk.nvme0n1.out‚ 8525.83984375 disk.dm-1.out‚ 165.515625 disk.drbd2265.out‚0 network.recv‚ 1.268072e+06 disk.nvme0n1.in‚ 16.6953125 disk.dm-0.out‚ 8525.83984375 disk.sda.in‚0 disk.sda.out‚ 165.515625 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 165.515625 disk.drbd2266.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 569.33203125 proc.cpu.threads‚17sÏ$æ2¬- _wandbruntime‚3540  _runtime‚3540é„? É­Ã ' lr-AdamW/pg1‚0.00021427697258391874 ' lr-AdamW/pg2‚0.00021427697258391874 trainer/global_step‚4599 % _timestamp‚1.7884084378463776e+09 _runtime‚3540.833866187 _step‚373%£ kÎ®È ' lr-AdamW/pg1‚0.00021427697258391874 ' lr-AdamW/pg2‚0.00021427697258391874 trainer/global_step‚4599 % _timestamp‚1.7884084378463776e+09 _runtime‚3540.833866187 _step‚373õ¸½¬2¯- _wandbruntime‚3540  _runtime‚3540pyÇmH°×O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66523530 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60684254 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290772562198016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259245221547008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884084378470957e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60683807 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚4599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeßÜõ•!viceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4028041073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259245204507648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294800417804288 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270041251757056 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86577087 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚266201748626944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3840159492608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3840159492608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294800603271168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3839503130112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21876051 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263071248618894 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3826027071886 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66523532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266201766629888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86597125 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153120657 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153100617 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270041926122496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4027879546368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3839503130112 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86597125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263070577141778 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21876496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚3540.834695532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66523530 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38807758 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2700419261224967iƒ0â'±ÜO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66523530 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38807758 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚270041926122496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66523530 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60684254 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290772562198016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259245221547008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884084378470957e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60683807 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚4599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4028041073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259245204507648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294800417804288 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270041251757056 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86577087 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚266201748626944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3840159492608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3840159492608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294800603271168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3839503130112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21876051 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263071248618894 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3826027071886 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66523532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266201766629888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86597125 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153120657 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153100617 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270041926122496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4027879546368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3839503130112 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86597125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263070577141778 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21876496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚3540.834695532ö(›.X2²- _wandbruntime‚3541  _runtime‚3541v’¡þç&³áM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3840978933760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38815924 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚3541.522211409 _step‚375 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 trainer/global_step‚4599 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66537548 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270099098602496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚266258101665792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86615605 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86595541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266258101665792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4028901637632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60696945 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38815926 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270099098602496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60696503 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290834188995584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86615605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21880579 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66537546 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266258119668736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290834212935680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826189069094 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153133087 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3840322557952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4028738027008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66537546 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884084385341716e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270098424223744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3840978933760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ðµr³M´æM P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 trainer/global_step‚4599 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66537548 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270099098602496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚266258101665792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86615605 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86595541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266258101665792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4028901637632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60696945 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38815926 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270099098602496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60696503 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290834188995584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86615605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21880579 J=DeviceStatsMonitor.on_train_batch_end/âyqŸactive.large_pool.freed‚66537546 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266258119668736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290834212935680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826189069094 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153133087 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3840322557952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4028738027008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66537546 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884084385341716e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270098424223744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3840978933760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3840978933760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38815924 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚3541.522211409 _step‚375÷ÓZDB2µ- _wandbruntime‚3541  _runtime‚3541k¡¹ÄX¶Ò 1loss/SRqqt4/detection_loss‚1.6697401316570384 /loss/FRt1/assignment_loss‚4.274708766937255 2loss/SRqqt2/assignment_loss‚1.9594043528830463 0loss/FBt1/assignment_loss‚0.2966165518025394 1loss/FBt3/assignment_loss‚0.12685652795474925 .loss/FBt2/detection_loss‚3.379690067635642 /loss/FRt2/assignment_loss‚3.620752258998592 0loss/SRqqt1/detection_loss‚2.438623386952612 /loss/FBt3/detection_loss‚0.5985875287849922 trainer/global_step‚4599 0loss/FRt4/assignment_loss‚2.3852203645796144 0loss/SRqqt4/assignment_loss‚1.33024883955451 /loss/FRt2/detection_loss‚2.2550564611830413 /loss/FBt4/detection_loss‚1.7084943226405553 epoch‚3 /loss/FRt3/assignment_loss‚1.791727903816435 2loss/SRqqt1/assignment_loss‚1.5001934717098873 2loss/SRqqt3/assignment_loss‚1.0293745289816232 /loss/FRt1/detection_loss‚3.2178950929641723 .loss/FRt4/detection_loss‚2.187647090205606 1loss/FBt4/assignment_loss‚0.40136106096075047 /loss/FRt3/detection_loss‚1.6335054840892553 1loss/SRqqt3/detection_loss‚1.2164997289293125 .loss/FBt1/detection_loss‚1.949166836483138 % _timestamp‚1.7884084385362837e+09 _runtime‚3541.523646557 _step‚376 1loss/FBt2/assignment_loss‚0.39208424562821165 0loss/SRqqt2/detection_loss‚1.952832700853998 %loss/total_loss‚43.31628770618658´{u]·× /loss/FBt4/detection_loss‚1.7084943226405553 epoch‚3 /loss/FRt3/assignment_loss‚1.791727903816435 2loss/SRqqt1/assignment_loss‚1.5001934717098873 2loss/SRqqt3/assignment_loss‚1.0293745289816232 /loss/FRt1/detection_loss‚3.2178950929641723 .loss/FRt4/detection_loss‚2.187647090205606 1loss/FBt4/assignment_loss‚0.40136106096075047 /loss/FRt3/detection_loss‚1.6335054840892553 1loss/SRqqt3/detection_loss‚1.2164997289293125 .loss/FBt1/detection_loss‚1.949166836483138 % _timestamp‚1.7884084385362837e+09 _runtime‚3541.523646557 _step‚376 1loss/FBt2/assignment_loss‚0.39208424562821165 0loss/SRqqt2/detection_loss‚1.952832700853998 %loss/total_loss‚43.31628770618658 1loss/SRqqt4/detection_loss‚1.6697401316570384 /loss/FRt1/assignment_loss‚4.274708766937255 2loss/SRqqt2/assignment_loss‚1.9594043528830463 0loss/FBt1/assignment_loss‚0.2966165518025394 1loss/FBt3/assignment_loss‚0.12685652795474925 .loss/FBt2/detection_loss‚3.379690067635642 /loss/FRt2/assignment_loss‚3.620752258998592 0loss/SRqqt1/detection_loss‚2.438623386952612 /loss/FBt3/detection_loss‚0.5985875287849922 trainer/global_step‚4599 0loss/FRt4/assignment_loss‚2.3852203645796144 0loss/SRqqt4/assignment_loss‚1.33024883955451 /loss/FRt2/detection_loss‚2.2550564611830413øuø¬î‘¸‚(:† „åãÔÖ¥ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.973 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñïõ V¹‚(:Ë „åãÔŠÛ  disk.drbd2266.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 16.6953125 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2720.20703125 network.sent‚ 1.0269752e+07 network.recv‚ 1.271866e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚8526.60693359375 disk.dm-1.out‚ 165.97265625 disk.drbd2080.in‚0$ memory_percent‚96.52795791625977 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚ 16.6953125" disk.dm-0.out‚8526.60693359375 disk.drbd2265.out‚0 disk.sda.out‚ 165.97265625 disk.rbd1.out‚17.75 disk.nvme0n1.in‚ 16.6953125% disk./.usageGB‚232.73297119140625 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚8526.60693359375 disk.sda3.in‚0 disk.drbd2266.in‚0& proc.memory.availableMB‚ 568.859375) proc.memory.percent‚16.60282611846924 cpu‚25.304180798325973 proc.cpu.threads‚17 disk.drbd2080.out‚0* disk./.usagePercent‚15.618904972624806 disk.sda3.out‚ 165.97265625µ:4’º‚(:‡ “åãԺɓ gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ßJ»‚(:¿ “åãÔë’Ó  disk.drbd2266.out‚0$ memory_percent‚96.52626514434814 network.recv‚ 1.278288e+06 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0) proc.memory.percent‚16.60282611846924 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 8545.35546875 disk.rbd1.in‚0 disk.dm-0.out‚ 8545.35546875 disk.dm-1.out‚ 169.65234375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0" disk.nvme0n1.out‚ 8545.35546875 disk.drbd2265.in‚0 disk.sda.out‚ 169.65234375 disk.drbd2080.in‚0 disk.rbd0.in‚0% disk./.usageGB‚232.73682022094727 disk.sda1.out‚0 disk.sda3.out‚ 169.65234375( proc.memory.availableMB‚ 569.13671875# proc.memory.rssMB‚ 2720.20703125* disk./.usagePercent‚15.619163284226856 cpu‚25.30312565403484 network.sent‚ 1.0378947e+07 disk.drbd1716.in‚0 disk.rbd1.out‚17.8125 disk.nvme0n1.in‚16.71875 disk.dm-0.in‚16.71875 disk.nvme0c0n1.in‚16.71875 disk.dm-1.in‚0 disk.drbd2266.in‚0ÎúèC2¼- _wandbruntime‚3576  _runtime‚3576†WŸ2ɽà % _timestamp‚1.7884084739217923e+09 _runtime‚3576.909253679 _step‚377 ' lr-AdamW/pg1‚0.00021262152836734355 ' lr-AdamW/pg2‚0.00021262152836734355 trainer/global_step‚4649ú¾}Î¾È _step‚377 ' lr-AdamW/pg1‚0.00021262152836734355 ' lr-AdamW/pg2‚0.00021262152836734355 trainer/global_step‚4649 % _timestamp‚1.7884084739217923e+09 _runtime‚3576.909253679ùºq 2¿- _wandbruntime‚3576  _runtime‚3576mF’Ý'À×O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67224332 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265856619009186 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61319454 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67224332 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3866862411294 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22102878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61319034 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87523177 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67224330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297925962301440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87523177 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22103296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3866207973538 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269019418572288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚3576.910199363 _step‚378 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269019400569344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87503139 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3880488935936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297925776834560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261990428075008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269019418572288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272899889505280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265857290486302 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216156 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39216158 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154747509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088]É¿ó}'ÁÜO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67224332 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265856619009186 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61319454 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67224332 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3866862411294 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22102878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61319034 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87523177 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67224330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297925962301440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87523177 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22103296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3866207973538 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269019418572288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚3576.910199363 _step‚378 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269019400569344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87503139 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3880488935936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297925776834560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261990428075008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269019418572288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272899889505280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265857290486302 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216156 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39216158 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154747509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 6544=Ìäe37756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928úk€2Â- _wandbruntime‚3577  _runtime‚3577>‚™yç&ÃáM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262045315166208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272957061971968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22107408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4071725927424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39224324 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269075753608192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293916749818880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3881964739584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154759939 _step‚379 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61331732 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293916725878784 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3881308363776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39224326 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87521593 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3867678846466 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297988292389888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269075771611136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3867024408502 % _timestamp‚1.7884084745971134e+09 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61332168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262045332205568 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154780005 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚272957736350720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67238346 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚4649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87521593 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4071566511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87541657 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269075771611136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67238348 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269075753608192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67238348 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297988475746304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414¥(.Kì&ÄæM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272957061971968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22107408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4071725927424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39224324 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269075753608192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293916749818880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3881964739584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154759939 _step‚379 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61331732 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293916725878784 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3881308363776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39224326 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87521593 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3867678846466 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297988292389888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269075771611136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3867024408502 % _timestamp‚1.7884084745971134e+09 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61332168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262045332205568 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154780005 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚272957736350720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67238346 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚4649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87521593 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4071566511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87541657 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269075771611136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67238348 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269075753608192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67238348 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297988475746304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262045315166208ûâªê2Å- _wandbruntime‚3577  _runtime‚3577E±õZÆÔ 1loss/FBt3/assignment_loss‚0.21079118275002892 0loss/FBt4/assignment_loss‚0.3167974086944014 /loss/FBt4/detection_loss‚1.9782503545284271 trainer/global_step‚4649 2loss/SRqqt4/assignment_loss‚1.4787718854708278 .loss/FRt2/detection_loss‚2.768133006989956 %loss/total_loss‚44.08392459281898 _step‚380 0loss/FRt2/assignment_loss‚3.3600437343120575 .loss/FRt3/assignment_loss‚1.58478466629982 2loss/SRqqt1/assignment_loss‚1.7441048589017656 0loss/FBt1/assignment_loss‚0.0761149395718978 /loss/FRt1/detection_loss‚2.8191754854086675 1loss/SRqqt1/detection_loss‚2.3159040303693876 .loss/FBt1/detection_loss‚2.525883436203003 2loss/SRqqt2/assignment_loss‚1.6043574685603381 .loss/FBt2/detection_loss‚3.267234537336561 % _timestamp‚1.7884084745985985e+09 0loss/FRt4/assignment_loss‚3.7416451342251835 /loss/FRt3/detection_loss‚1.6999832375347614 0loss/FRt1/assignment_loss‚3.4542454697869043 2loss/SRqqt3/assignment_loss‚0.9257771812952482 1loss/SRqqt2/detection_loss‚1.9243940269947053 0loss/FBt3/detection_loss‚0.46381964729702535 epoch‚3 _runtime‚3577.586330337 /loss/FBt2/assignment_loss‚0.318106960091326 .loss/FRt4/detection_loss‚2.422501702089699 1loss/SRqqt3/detection_loss‚1.0618835526017043 1loss/SRqqt4/detection_loss‚2.0212206855052854‚”¢Ü_ÇÙ 0loss/FRt1/assignment_loss‚3.4542454697869043 2loss/SRqqt3/assignment_loss‚0.9257771812952482 1loss/SRqqt2/detection_loss‚1.9243940269947053 0loss/FBt3/detection_loss‚0.46381964729702535 epoch‚3 _runtime‚3577.586330337 /loss/FBt2/assignment_loss‚0.318106960091326 .loss/FRt4/detection_loss‚2.422501702089699 1loss/SRqqt3/detection_loss‚1.0618835526017043 1loss/SRqqt4/detection_loss‚2.0212206855052854 1loss/FBt3/assignment_loss‚0.21079118275002892 0loss/FBt4/assignment_loss‚0.3167974086944014 /loss/FBt4/detection_loss‚1.9782503545284271 trainer/global_step‚4649 2loss/SRqqt4/assignment_loss‚1.4787718854708278 .loss/FRt2/detection_loss‚2.768133006989956 %loss/total_loss‚44.08392459281898 _step‚380 0loss/FRt2/assignment_loss‚3.3600437343120575 .loss/FRt3/assignment_loss‚1.58478466629982 2loss/SRqqt1/assignment_loss‚1.7441048589017656 0loss/FBt1/assignment_loss‚0.0761149395718978 /loss/FRt1/detection_loss‚2.8191754854086675 1loss/SRqqt1/detection_loss‚2.3159040303693876 .loss/FBt1/detection_loss‚2.525883436203003 2loss/SRqqt2/assignment_loss‚1.6043574685603381 .loss/FBt2/detection_loss‚3.267234537336561 % _timestamp‚1.7884084745985985e+09 0loss/FRt4/assignment_loss‚3.7416451342251835 /loss/FRt3/detection_loss‚1.6999832375347614ü`ˆ‰‘’È‚(:‡ ¢åãÔô­ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZžôMÉ‚(: ¢åãÔ¾ö¨  disk.sda1.in‚0# proc.memory.rssMB‚ 2720.20703125 disk.dm-0.in‚16.71875 disk.dm-1.out‚ 170.27734375 disk.sda2.out‚0& proc.memory.availableMB‚ 568.359375 disk.drbd2080.out‚0% disk./.usageGB‚232.73783493041992 disk.drbd1716.in‚0 disk.rbd1.out‚17.90625 disk.sda.in‚0 disk.sda3.out‚ 170.27734375 disk.drbd2080.in‚0% disk.nvme0c0n1.out‚8551.970703125* disk./.usagePercent‚15.619231382230964 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.302064036632068 proc.cpu.threads‚17# disk.nvme0n1.out‚8551.970703125 disk.sda.out‚ 170.27734375 network.sent‚ 1.0388257e+07 disk.rbd1.in‚0 disk.dm-0.out‚8551.970703125 disk.sda3.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚16.71875 disk.nvme0n1.in‚16.71875 disk.dm-1.in‚0) proc.memory.percent‚16.60282611846924 network.recv‚ 1.289354e+06 disk.drbd2266.out‚0 disk.rbd0.in‚0$ memory_percent‚96.53100967407227 disk.drbd2265.out‚0 disk.drbd2266.in‚0"¿:G’Ê‚(:‡ ±åãÔäÉø gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.818' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1636 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½`#ÍNË‚(:à ±åãÔœ á % disk./.usageGB‚232.73773956298828 disk.dm-1.out‚ 171.01171875 disk.rbd0.in‚0$ memory_percent‚96.53036594390869# proc.memory.rssMB‚ 2720.20703125 disk.dm-0.in‚16.71875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 568.46484375 proc.cpu.threads‚17 network.recv‚ 1.295783e+06 disk.dm-0.out‚8552.326171875 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.rbd1.out‚17.96875# disk.nvme0n1.out‚8552.326171875 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚8552.326171875 disk.drbd1716.in‚0 network.sent‚ 1.0497601e+07* disk./.usagePercent‚15.619224982042606 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 171.01171875 disk.nvme0c0n1.in‚16.71875 disk.nvme0n1.in‚16.71875) proc.memory.percent‚16.60282611846924 cpu‚25.30101484737536 disk.sda.out‚ 171.01171875 disk.drbd2266.in‚0 disk.rbd1.in‚0övŒþ2Ì- _wandbruntime‚3613  _runtime‚3613UkøßÈÍ ' lr-AdamW/pg1‚0.00021095043508979818 ' lr-AdamW/pg2‚0.00021095043508979818 trainer/global_step‚4699 $ _timestamp‚1.788408510253293e+09 _runtime‚3613.240729178 _step‚381Œû|ÆÍÎÇ _step‚381 ' lr-AdamW/pg1‚0.00021095043508979818 ' lr-AdamW/pg2‚0.00021095043508979818 trainer/global_step‚4699 $ _timestamp‚1.788408510253293e+09 _runtime‚3613.240729178ý‚´2Ï- _wandbruntime‚3613  _runtime‚3613¢=Ì8ƒÐ×O cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4113864276992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268642660876594 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264735617563648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88429191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚4699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88449229 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61954468 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22329910 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301051500241408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275759201618944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3907043312946 _step‚382 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67925130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88449229 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22329462 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264735634603008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275758527253504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296937635964416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156374361 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275758527253504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301051312677376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3921474741760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3922131104256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268643332353710 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271837052511744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275759201618944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67925132 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128¸.'…â'ÑÜO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67925130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88449229 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22329462 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264735634603008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275758527253504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296937635964416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156374361 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275758527253504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301051312677376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3921474741760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3922131104256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268643332353710 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271837052511744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275759201618944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67925132 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4113864276992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268642660876594 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264735617563648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88429191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚4699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88449229 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61954468 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22329910 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301051500241408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275759201618944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3907043312946 _step‚382 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0þ})æR2Ò- _wandbruntime‚3613  _runtime‚3613ën›ç&ÓáM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268698381442118 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3922294169600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88467709 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271893423553536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚383 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61967089 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3922950545408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275816374098944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301113828401664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296999262761984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4114727153152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4114565639680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156406857 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67939148 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39632726 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3907859747910 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22334363 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61966649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884085109319913e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275816374098944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296999286702080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3908514185874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3613.919870662 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271893405550592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271893405550592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264790538733568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚4699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88447645 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22334363 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61966649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884085109319913e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275816374098944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296999286702080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3908514185874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3613.919870662 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271893405550592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271893405550592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264790538733568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚4699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88447645 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88467709 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271893423553536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚383 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61967089 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3922950545408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275816374098944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301113828401664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296999262761984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_àýýdend/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4114727153152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4114565639680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156406857 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67939148 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39632726 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3907859747910ÿ†èŒH2Õ- _wandbruntime‚3613  _runtime‚3613¢à+NYÖÓ 2loss/SRqqt2/assignment_loss‚1.7242199099949889 1loss/FBt1/assignment_loss‚0.21079532207325732 .loss/FRt2/detection_loss‚2.137583063678755 1loss/SRqqt4/detection_loss‚2.0172256296500564 /loss/FBt3/detection_loss‚0.6087073631076613 %loss/total_loss‚45.06787183097116 $ _timestamp‚1.788408510933925e+09 0loss/FRt2/assignment_loss‚3.9101291753743825 /loss/FRt4/assignment_loss‚3.211838027705317 1loss/FBt2/assignment_loss‚0.27775964359752836 1loss/SRqqt2/detection_loss‚2.2846316944908454 1loss/SRqqt3/detection_loss‚0.8723197012208402 1loss/SRqqt3/assignment_loss‚0.934552010067273 /loss/FBt1/detection_loss‚3.1412150313456855 /loss/FRt3/assignment_loss‚1.398489254301992 /loss/FRt1/detection_loss‚3.1879872168813432 1loss/SRqqt1/assignment_loss‚2.064716726541519 2loss/SRqqt4/assignment_loss‚1.5600144863627585 1loss/FBt3/assignment_loss‚0.16255157716425672 /loss/SRqqt1/detection_loss‚2.67210062344869 /loss/FBt4/detection_loss‚2.5420357414654324 _runtime‚ 3613.92128994 _step‚384 0loss/FRt1/assignment_loss‚3.3960989713668823 /loss/FRt3/detection_loss‚1.3767324987927387 epoch‚3 1loss/FBt4/assignment_loss‚0.15879397450147995 /loss/FBt2/detection_loss‚2.7805471152067183 .loss/FRt4/detection_loss‚2.436827072630758 trainer/global_step‚4699âÎCµ^×Ø 1loss/SRqqt4/detection_loss‚2.0172256296500564 /loss/FBt3/detection_loss‚0.6087073631076613 %loss/total_loss‚45.06787183097116 $ _timestamp‚1.788408510933925e+09 0loss/FRt2/assignment_loss‚3.9101291753743825 /loss/FRt4/assignment_loss‚3.211838027705317 1loss/FBt2/assignment_loss‚0.27775964359752836 1loss/SRqqt2/detection_loss‚2.2846316944908454 1loss/SRqqt3/detection_loss‚0.8723197012208402 1loss/SRqqt3/assignment_loss‚0.934552010067273 /loss/FBt1/detection_loss‚3.1412150313456855 /loss/FRt3/assignment_loss‚1.398489254301992 /loss/FRt1/detection_loss‚3.1879872168813432 1loss/SRqqt1/assignment_loss‚2.064716726541519 2loss/SRqqt4/assignment_loss‚1.5600144863627585 1loss/FBt3/assignment_loss‚0.16255157716425672 /loss/SRqqt1/detection_loss‚2.67210062344869 /loss/FBt4/detection_loss‚2.5420357414654324 _runtime‚ 3613.92128994 _step‚384 0loss/FRt1/assignment_loss‚3.3960989713668823 /loss/FRt3/detection_loss‚1.3767324987927387 epoch‚3 1loss/FBt4/assignment_loss‚0.15879397450147995 /loss/FBt2/detection_loss‚2.7805471152067183 .loss/FRt4/detection_loss‚2.436827072630758 trainer/global_step‚4699 2loss/SRqqt2/assignment_loss‚1.7242199099949889 1loss/FBt1/assignment_loss‚0.21079532207325732 .loss/FRt2/detection_loss‚2.137583063678755€ÔC#¸Ø‚(:’ ÀåãÔç° gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚139.396' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.879199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 [:KPÙ‚(:Å ÀåãÔô«Í  disk.rbd0.out‚0 network.sent‚ 1.0502836e+07 disk.sda3.out‚ 171.55078125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚17.96875$ memory_percent‚96.52879238128662 proc.cpu.threads‚17 disk.dm-1.out‚ 171.55078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.recv‚ 1.299426e+06% disk./.usageGB‚232.73773956298828& disk.nvme0c0n1.out‚8553.3076171875( proc.memory.availableMB‚ 568.72265625 disk.drbd2080.out‚0 disk.sda1.out‚0 cpu‚25.2999681688664 disk.sda2.in‚0 disk.nvme0c0n1.in‚16.71875$ disk.nvme0n1.out‚8553.3076171875 disk.sda.in‚0 disk.sda.out‚ 171.55078125 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚16.71875! disk.dm-0.out‚8553.3076171875) proc.memory.percent‚16.60282611846924* disk./.usagePercent‚15.619224982042606 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2720.20703125 disk.dm-0.in‚16.71875 disk.rbd0.in‚0‰(™ýÚ‚(:… ÏåãÔ¯ýœ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„àE[Û‚(:Ð ÏåãÔÌå  disk.drbd2080.in‚0( proc.memory.availableMB‚ 569.17578125# proc.memory.rssMB‚ 2720.20703125 disk.sda.in‚0 disk.drbd2080.out‚0% disk.nvme0c0n1.out‚8753.169921875 disk.rbd1.in‚0 disk.dm-0.in‚ 17.11328125 disk.sda.out‚ 172.41015625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 17.11328125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 17.11328125 disk.drbd1716.out‚0 disk.rbd1.out‚ 18.0009765625# disk.nvme0n1.out‚8753.419921875$ disk./.usageGB‚236.3978157043457 network.recv‚ 1.306427e+06 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚25.29890001196495 proc.cpu.threads‚17 network.sent‚ 1.0612485e+07 disk.sda3.out‚ 172.41015625 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 172.41015625* disk./.usagePercent‚15.864855762897538 disk.dm-0.out‚8745.919921875$ memory_percent‚96.52602672576904) proc.memory.percent‚16.60282611846924¡jjØ’Ü‚(:‡ ÞåãÔ À¹ gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌâÍÀaÝ‚(:Ö ÞåãÔ Èß  disk.dm-0.in‚ 26.9765625( disk.nvme0c0n1.out‚12630.02294921875$ memory_percent‚96.52853012084961) proc.memory.percent‚16.60282611846924 cpu‚25.297883280058276* disk./.usagePercent‚15.886246984436053 disk.nvme0n1.in‚ 26.9765625 disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda.out‚ 173.03515625 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 173.03515625# proc.memory.rssMB‚ 2720.20703125 disk.rbd1.out‚ 18.0947265625 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 network.sent‚ 1.0622137e+07& disk.nvme0n1.out‚12630.02294921875# disk.dm-0.out‚12630.02294921875 disk.sda3.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 network.recv‚ 1.318139e+06 disk.rbd0.out‚0& proc.memory.availableMB‚ 568.765625% disk./.usageGB‚236.71656036376953 disk.dm-1.out‚ 173.03515625 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 26.9765625#/Çã2Þ- _wandbruntime‚3650  _runtime‚3650b QCÉßà ' lr-AdamW/pg1‚0.00020926404100461685 ' lr-AdamW/pg2‚0.00020926404100461685 trainer/global_step‚4749 % _timestamp‚1.7884085471898196e+09 _runtime‚3650.177276427 _step‚385ÌBÎàÈ ' lr-AdamW/pg1‚0.00020926404100461685 ' lr-AdamW/pg2‚0.00020926404100461685 trainer/global_step‚4749 % _timestamp‚1.7884085471898196e+09 _runtime‚3650.177276427 _step‚385áû*2á- _wandbruntime‚3650  _runtime‚3650C„¾ŒÝ'â×O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68625930 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274654704454144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40032956 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22556241 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4156807825920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271428702744002 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89355243 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274654722457088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884085471905513e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278617165001728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚386 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22556686 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157981173 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89355243 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4156646299136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3963116910080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304176980673536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62589197 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278617839367168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62589644 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3963116910080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274654722457088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962460547584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158001213 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3948533090110 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89375281 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278617165001728 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚4749 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3947878652354 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304176795206656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884085471905513e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278617165001728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚386 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22556686 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157981173 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89355243 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4156646299136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3963116910080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304176980673536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62589197 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278617839367168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62589644 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3963116910080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274654722457088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962460547584‚œxPõ2ä- _wandbruntime‚3650  _runtime‚3650ðäe¶~ åáM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963936351232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274711075495936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3963936351232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278674337468416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300081799645184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89393761 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271485094786850 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62601824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278675011847168 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚387 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89373697 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚4749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89373697 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22560700 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4157490690048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304239490468864 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22561155 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267535745261568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158013643 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158033709 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278675011847168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271484423309526 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274711057492992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3949349525282 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3963279975424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157666883584 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68639946 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40041126 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267535728222208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68639948 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948695087318 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68639948 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68639946 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89393761 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908ƒ¶Øì&ææM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89393761 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271485094786850 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62601824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278675011847168 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚387 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89373697 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚4749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89373697 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22560700 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4157490690048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304239490468864 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22561155 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267535745261568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158013643 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158033709 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278675011847168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271484423309526 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274711057492992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3949349525282 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3963279975424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157666883584 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68639946 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40041126 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267535728222208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68639948 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948695087318 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68639948 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68639946 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89393761 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963936351232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274711075495936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3963936351232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278674337468416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300081799645184ƒèÉ¿2ç- _wandbruntime‚3650  _runtime‚3650œ&ö\èÖ epoch‚3 trainer/global_step‚4749 _step‚388 2loss/SRqqt4/assignment_loss‚1.5084440212468713 1loss/FBt2/assignment_loss‚0.38756241591181606 1loss/SRqqt1/detection_loss‚2.3073231385600184 .loss/FBt1/detection_loss‚1.825964337045496 0loss/FRt3/assignment_loss‚1.7207391084508692 2loss/SRqqt2/assignment_loss‚1.7424686873564497 .loss/FRt1/assignment_loss‚3.84824948238604 1loss/SRqqt2/detection_loss‚1.5976396028418094 /loss/FBt3/detection_loss‚0.6777514418587088 .loss/FRt2/detection_loss‚2.090430296129651 2loss/SRqqt3/assignment_loss‚0.9483241385745781 1loss/SRqqt3/detection_loss‚1.2321556890728298 .loss/FBt2/detection_loss‚2.523526594042778 .loss/FRt1/detection_loss‚2.791523595650991 0loss/FRt2/assignment_loss‚3.3701801127857633 0loss/FRt4/assignment_loss‚2.9996176442857516 1loss/FBt3/assignment_loss‚0.19103742748795896 4loss/FBt4/assignment_loss‚0.00012730741582345217 /loss/FRt3/detection_loss‚1.6474662924066503 1loss/SRqqt4/detection_loss‚1.6846083992598009 %loss/total_loss‚47.33144135202839 /loss/FBt1/assignment_loss‚0.355207606007091 .loss/FRt4/detection_loss‚2.335045874118805 .loss/FBt4/detection_loss‚7.180654764175415 % _timestamp‚1.7884085478847198e+09 _runtime‚3650.872093374 2loss/SRqqt1/assignment_loss‚2.3653933749564233â‚ï7aéÛ 1loss/SRqqt3/detection_loss‚1.2321556890728298 .loss/FBt2/detection_loss‚2.523526594042778 .loss/FRt1/detection_loss‚2.791523595650991 0loss/FRt2/assignment_loss‚3.3701801127857633 0loss/FRt4/assignment_loss‚2.9996176442857516 1loss/FBt3/assignment_loss‚0.19103742748795896 4loss/FBt4/assignment_loss‚0.00012730741582345217 /loss/FRt3/detection_loss‚1.6474662924066503 1loss/SRqqt4/detection_loss‚1.6846083992598009 %loss/total_loss‚47.33144135202839 /loss/FBt1/assignment_loss‚0.355207606007091 .loss/FRt4/detection_loss‚2.335045874118805 .loss/FBt4/detection_loss‚7.180654764175415 % _timestamp‚1.7884085478847198e+09 _runtime‚3650.872093374 2loss/SRqqt1/assignment_loss‚2.3653933749564233 epoch‚3 trainer/global_step‚4749 _step‚388 2loss/SRqqt4/assignment_loss‚1.5084440212468713 1loss/FBt2/assignment_loss‚0.38756241591181606 1loss/SRqqt1/detection_loss‚2.3073231385600184 .loss/FBt1/detection_loss‚1.825964337045496 0loss/FRt3/assignment_loss‚1.7207391084508692 2loss/SRqqt2/assignment_loss‚1.7424686873564497 .loss/FRt1/assignment_loss‚3.84824948238604 1loss/SRqqt2/detection_loss‚1.5976396028418094 /loss/FBt3/detection_loss‚0.6777514418587088 .loss/FRt2/detection_loss‚2.090430296129651 2loss/SRqqt3/assignment_loss‚0.9483241385745781„gbiñ’ê‚(:‡ íåãÔ–Ý‹ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ %B^ë‚(:Ó íåãÔÆâ  disk.dm-0.in‚ 29.2265625$ memory_percent‚96.52867317199707 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda2.in‚0" disk.dm-0.out‚12634.4443359375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 29.2265625' proc.memory.availableMB‚ 568.7421875% disk./.usageGB‚236.71836853027344 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 29.2265625% disk.nvme0n1.out‚12634.4443359375 disk.drbd2265.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2720.20703125* disk./.usagePercent‚15.886368332007283 disk.dm-1.out‚ 173.89453125 disk.drbd2080.out‚0 cpu‚25.296806169181444' disk.nvme0c0n1.out‚12634.4443359375 disk.sda.out‚ 173.89453125 disk.drbd2266.in‚0) proc.memory.percent‚16.60282611846924 disk.rbd1.out‚ 18.2431640625 disk.sda3.in‚0 disk.sda3.out‚ 173.89453125 disk.drbd2080.in‚0 network.sent‚ 1.062824e+07 network.recv‚ 1.322081e+06 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0rŽèçì‚(:… üåãÔй– gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚140.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºœ]^í‚(:Ó üåãÔËü«  disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 29.2265625 disk.drbd1716.out‚0% disk.nvme0n1.out‚12635.1005859375& proc.memory.availableMB‚ 568.734375) proc.memory.percent‚16.60282611846924 disk.sda3.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚0 network.sent‚ 1.0735379e+07* disk./.usagePercent‚15.886255688692216' disk.nvme0c0n1.out‚12635.1005859375 disk.dm-0.in‚ 29.2265625 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda.in‚0$ memory_percent‚96.52872085571289# proc.memory.rssMB‚ 2720.20703125 cpu‚25.295744083323893 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 29.2265625" disk.dm-0.out‚12635.1005859375 proc.cpu.threads‚17 network.recv‚ 1.328409e+06 disk.drbd2266.out‚0 disk.sda3.out‚ 174.38671875 disk.rbd0.in‚0 disk.dm-1.out‚ 174.38671875 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 18.2431640625 disk.drbd2265.in‚0 disk.sda.out‚ 174.38671875 disk.sda1.in‚0% disk./.usageGB‚236.71669006347656…Iá2î- _wandbruntime‚3686  _runtime‚3686íš;Åï¿ & lr-AdamW/pg1‚0.0002075626975537994 & lr-AdamW/pg2‚0.0002075626975537994 trainer/global_step‚4799 # _timestamp‚1.78840858311763e+09 _runtime‚3686.105101757 _step‚389†¡ŒÊðÄ _runtime‚3686.105101757 _step‚389 & lr-AdamW/pg1‚0.0002075626975537994 & lr-AdamW/pg2‚0.0002075626975537994 trainer/global_step‚4799 # _timestamp‚1.78840858311763e+09…ƒõ{Â2ñ- _wandbruntime‚3686  _runtime‚3686<8ßl‰$ò×O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69326732 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40441356 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307302499436032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281475802749952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884085831183558e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159608025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281475802749952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69326732 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303102685790720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307302311872000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90281295 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4003446353408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3989368429518 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277472356396544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4199626081280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40441358 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4199789705216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚390 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277472374399488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚277472356396544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69326730 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63225733 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274215416088526 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274214744611410 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69326730 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22783943 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22784375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281476477115392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90281295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4004102715904 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22784375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281476477115392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90281295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4004102715904 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281475802749952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884085831183558e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159608025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281475802749952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69326732 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303102685790720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307302311872000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90281295 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4003446353408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3989368429518 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277472356396544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4199626081280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40441358 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4199789705216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚390 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277472374399488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚277472356396544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69326730 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63225733 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274215416088526 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274214744611410 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69326730 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22783943 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4200654050816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4004922157056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40449524 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307364824871424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4200488343040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281533649595392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚277528709435392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303164336528384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281532975216640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69340748 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69340748 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90299749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4004265781248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274270465176934 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40449526 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270280951789568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚270280934750208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159640495 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4799 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277528727438336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303164360468480 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚391 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4004922157056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63237945 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281533649595392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚277528727438336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22788421 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990184864690 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884085837961223e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90319813 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚277528709435392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69340748 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69340748 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90299749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4004265781248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274270465176934 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40449526 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270280951789568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚270280934750208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159640495 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4799 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277528727438336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303164360468480 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚391 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4004922157056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63237945 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281533649595392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚277528727438336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22788421 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990184864690 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884085837961223e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90319813 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚277528709435392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4200654050816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4004922157056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40449524 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307364824871424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4200488343040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281533649595392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0‡kIW2÷- _wandbruntime‚3686  _runtime‚3686Ï‘ÝÑ[øÕ /loss/FBt2/detection_loss‚2.3221928639845415 .loss/FRt1/detection_loss‚3.430762007122948 /loss/FRt2/assignment_loss‚3.053380157730796 2loss/SRqqt3/assignment_loss‚0.8874583812299329 1loss/SRqqt2/detection_loss‚1.7564638453834462 _step‚392 /loss/FBt3/detection_loss‚0.8663788045374438 2loss/SRqqt4/assignment_loss‚1.5502148246842211 /loss/FRt4/detection_loss‚2.1404488010839984 1loss/SRqqt1/detection_loss‚2.3698325044399984 % _timestamp‚1.7884085837976015e+09 0loss/FBt1/assignment_loss‚0.6348224556022615 1loss/FBt4/assignment_loss‚0.18919507082318887 epoch‚3 /loss/FRt3/assignment_loss‚1.238349669119891 1loss/SRqqt1/assignment_loss‚2.047267347574234 /loss/FBt1/detection_loss‚3.2526861971074883 %loss/total_loss‚44.97232638559913 /loss/FRt4/assignment_loss‚3.049561699953946 0loss/FBt3/assignment_loss‚0.1825185415122507 /loss/FRt3/detection_loss‚1.8439238957476383 1loss/SRqqt4/detection_loss‚1.6520304985601326 /loss/FBt4/detection_loss‚3.5218796730041504 trainer/global_step‚4799 _runtime‚3686.785290834 0loss/FRt1/assignment_loss‚3.4744220972061157 2loss/SRqqt2/assignment_loss‚1.5936573286593523 0loss/FBt2/assignment_loss‚0.8455393729646775 1loss/SRqqt3/detection_loss‚0.9912433996796608 /loss/FRt2/detection_loss‚2.07809694788681256²×ùÚ 2loss/SRqqt4/assignment_loss‚1.5502148246842211 /loss/FRt4/detection_loss‚2.1404488010839984 1loss/SRqqt1/detection_loss‚2.3698325044399984 % _timestamp‚1.7884085837976015e+09 0loss/FBt1/assignment_loss‚0.6348224556022615 1loss/ô(`FBt4/assignment_loss‚0.18919507082318887 epoch‚3 /loss/FRt3/assignment_loss‚1.238349669119891 1loss/SRqqt1/assignment_loss‚2.047267347574234 /loss/FBt1/detection_loss‚3.2526861971074883 %loss/total_loss‚44.97232638559913 /loss/FRt4/assignment_loss‚3.049561699953946 0loss/FBt3/assignment_loss‚0.1825185415122507 /loss/FRt3/detection_loss‚1.8439238957476383 1loss/SRqqt4/detection_loss‚1.6520304985601326 /loss/FBt4/detection_loss‚3.5218796730041504 trainer/global_step‚4799 _runtime‚3686.785290834 0loss/FRt1/assignment_loss‚3.4744220972061157 2loss/SRqqt2/assignment_loss‚1.5936573286593523 0loss/FBt2/assignment_loss‚0.8455393729646775 1loss/SRqqt3/detection_loss‚0.9912433996796608 /loss/FRt2/detection_loss‚2.0780969478868125 /loss/FBt2/detection_loss‚2.3221928639845415 .loss/FRt1/detection_loss‚3.430762007122948 /loss/FRt2/assignment_loss‚3.053380157730796 2loss/SRqqt3/assignment_loss‚0.8874583812299329 1loss/SRqqt2/detection_loss‚1.7564638453834462 _step‚392 /loss/FBt3/detection_loss‚0.8663788045374438ˆ}^¦ý’ú‚(:‡ ‹æãÔ½¸ª gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»¼Éd\û‚(:Ñ ‹æãÔ‚€® ( proc.memory.availableMB‚ 568.66796875 cpu‚25.294723347212397* disk./.usagePercent‚15.886255688692216 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0! disk.dm-0.out‚12636.193359375 disk.dm-1.out‚ 174.91015625 disk.drbd2265.out‚0 network.sent‚ 1.0742059e+07 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 29.2265625 disk.drbd1716.out‚0 disk.dm-0.in‚ 29.2265625 disk.sda.in‚0$ disk.nvme0n1.out‚12636.193359375 disk.sda3.out‚ 174.91015625$ memory_percent‚96.52912616729736) proc.memory.percent‚16.60282611846924 proc.cpu.threads‚17% disk./.usageGB‚236.71669006347656 disk.sda1.out‚0 disk.sda.out‚ 174.91015625& disk.nvme0c0n1.out‚12636.193359375 disk.rbd1.out‚ 18.275390625 disk.nvme0n1.in‚ 29.2265625 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2720.20703125 disk.sda2.in‚0 disk.drbd2266.out‚0 network.recv‚ 1.333461e+06 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.sda1.in‚0ŠÃ±ü‚(:… šæãÔ¬ ¬ gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.13' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C=*[ý‚(:Ð šæãÔ¡Ê  disk.drbd2265.in‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 568.42578125 disk.drbd2266.in‚0 disk.rbd1.in‚0# memory_percent‚96.5306043624878# proc.memory.rssMB‚ 2720.20703125) proc.memory.percent‚16.60282611846924 cpu‚25.293697218968887 disk.sda2.out‚0 disk.rbd1.out‚ 18.369140625 disk.nvme0n1.in‚ 29.2265625! disk.dm-0.out‚12636.478515625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.355579e+06* disk./.usagePercent‚15.886255688692216& disk.nvme0c0n1.out‚12636.478515625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 175.35546875 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚12636.478515625 disk.sda3.out‚ 175.35546875 disk.drbd2080.in‚0 disk.drbd1716.in‚0 network.sent‚ 1.0857145e+07% disk./.usageGB‚236.71669006347656 disk.nvme0c0n1.in‚ 29.2265625 disk.dm-0.in‚ 29.2265625 disk.dm-1.out‚ 175.35546875uX)œþ‚(:‘ ©æãÔª½å gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.38' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.675999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ ±_ÿ‚(:Ô ©æãÔšˆî  disk.rbd0.out‚0$ memory_percent‚97.02191352844238% disk./.usageGB‚236.71799850463867 disk.sda3.out‚ 177.85546875* disk./.usagePercent‚15.886343499276462 disk.drbd2266.out‚0 disk.drbd2080.out‚0& disk.nvme0n1.out‚12646.90771484375 disk.dm-0.in‚ 29.234375 disk.sda.in‚0 disk.sda.out‚ 177.85546875 disk.sda1.out‚0' proc.memory.availableMB‚ 487.9296875* proc.memory.percent‚17.094731330871582 disk.nvme0c0n1.in‚ 29.234375 disk.rbd1.in‚0 disk.sda2.in‚0 network.sent‚ 1.0864145e+07 disk.nvme0n1.in‚ 29.234375 disk.drbd2080.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 network.recv‚ 1.360977e+06( disk.nvme0c0n1.out‚12646.90771484375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 177.85546875# proc.memory.rssMB‚ 2800.80078125 cpu‚25.302196425183336 disk.rbd1.out‚ 18.431640625# disk.dm-0.out‚12646.90771484375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0án3Œ€‚(:’ ¸æãÔøŠ° gpu.0.gpu‚43.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚131.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.262400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÈ`8X‚(:Í ¸æãÔÁüÖ  disk.nvme0c0n1.in‚ 29.234375 disk.rbd1.out‚ 18.431640625 disk.nvme0n1.in‚ 29.234375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda3.out‚ 178.890625 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2800.8046875% disk./.usageGB‚236.71797943115234 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 178.890625 disk.sda2.out‚0$ memory_percent‚97.02191352844238* proc.memory.percent‚17.094755172729492 proc.cpu.threads‚17 disk.dm-0.in‚ 29.234375# disk.dm-0.out‚12647.06787109375* disk./.usagePercent‚15.886342219238788 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 network.recv‚ 1.364426e+06' proc.memory.availableMB‚ 487.9296875 cpu‚25.322161112466926& disk.nvme0n1.out‚12647.06787109375 network.sent‚ 1.0869438e+07 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚12647.06787109375 disk.drbd1716.in‚0 disk.dm-1.out‚ 178.890625 disk.rbd0.out‚0—Ú7»‚‚(:’ ÇæãÔÈåû gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.444' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.888800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÉ{S[ƒ‚(:Ð ÇæãÔãê  disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 29.234375 disk.sda.in‚0( proc.memory.availableMB‚ 488.96484375" disk.dm-0.out‚12650.3857421875 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 29.234375 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2800.8046875 network.sent‚ 1.0881854e+07 disk.sda3.out‚ 179.54296875 disk.dm-1.in‚0 disk.sda.out‚ 179.54296875 disk.dm-1.out‚ 179.54296875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 18.5263671875$ memory_percent‚97.01559543609619 network.recv‚ 1.381472e+06) disk./.usagePercent‚15.88634093920112 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0% disk.nvme0n1.out‚12650.3857421875 cpu‚25.34197085239042 disk.sda1.out‚0% disk./.usageGB‚236.71796035766602' disk.nvme0c0n1.out‚12650.3857421875 disk.drbd1716.in‚0 disk.dm-0.in‚ 29.234375* proc.memory.percent‚17.094755172729492ŽZk2„- _wandbruntime‚3760  _runtime‚3760X–|D*…¾T TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚70374818 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4052335142912 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚70374816 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚70374816 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚91327028 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4036925819398 trainer/global_step‚4835 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚91327028 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚285413932557824 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4051842312704 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161701844 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚64237866 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚41062270 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚278023803484678 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚273986877665280 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚281362108248064 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚3760.998386092 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚281362090245120 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4052335142912 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚70374818 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚285414443390976 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚281362108248064 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚307239697966080 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 161701844 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚273986894704640 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚23175444 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚307239674025984 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚23175596 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚154 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚311489438847488 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4249740881408 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41062268 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚285413932557824 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚311489299563520 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚91343005 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4249625537536 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161717823LçNI*†ÃT UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚91343005 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4249625537536 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161717823 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚70374818 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4052335142912 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚70374816 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚70374816 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚91327028 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4036925819398 trainer/global_step‚4835 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚91327028 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚285413932557824 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4051842312704 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161701844 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚64237866 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚41062270 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚278023803484678 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚273986877665280 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚281362108248064 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚3760.998386092 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚281362090245120 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4052335142912 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚70374818 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚285414443390976 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚281362108248064 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚307239697966080 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 161701844 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚273986894704640 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚23175444 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚307239674025984 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚23175596 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚154 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚311489438847488 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4249740881408 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41062268 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚285413932557824 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚311489299563520 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0‰¿ò£32‡- _wandbruntime‚3761  _runtime‚3761Y?¼$~ˆÇR UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚70385624 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚154 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚278059460853574 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4052700269568 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4037780343198 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚281398938276864 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 161719952 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91334330 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚307277549759488 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚285451127713280 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚281398920273920 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4052700269568 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚23180144 E?DeviceStatsMonitor.on_va!9Ï lidation_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4052207439360 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4835 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚64248764 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 161735931 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4052207439360 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚70385624 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4249990664192 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚281398938276864 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚23179992 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4250106008064 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41068772 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 _runtime‚3761.936349381 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚285451638546432 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚307277573699584 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚394 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚41068774 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚281398920273920 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚70385622 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚274022171424768 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚91334330 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 161735931 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚64248918 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚278059968807326 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161719952 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚91350307 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚285451638546432 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 $ _timestamp‚1.788408658948593e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚285451127713280 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4052207439360 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚70385624 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4249990664192 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚281398938276864 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚23179992 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4250106008064 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41068772 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 _runtime‚3761.936349381 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚285451638546432 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚307277573699584 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚394 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚41068774 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚281398920273920 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚70385622 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚274022171424768 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚91334330 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 161735931 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚64248918 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚278059968807326 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161719952 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚91350307 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚285451638546432 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 $ _timestamp‚1.788408658948593e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚285451127713280 B&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5107930302619934 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 epoch‚3 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 *jet/accuracy_1_of_3‚0.8627812266349792 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.49161630868911743 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019806981086730957 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 *jet/accuracy_3_of_4‚0.0641067773103714 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.163640096783638 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4951401948928833 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9983371496200562 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_4‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5388551354408264 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666567325592 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14190487563610077 particle/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5336984395980835 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9905575513839722 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1450464129447937 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4289851486682892 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4889158010482788 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03929402306675911 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09259707480669022 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9773243069648743 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3023809790611267 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4303557872772217 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.191361203789711 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 +particle/specificity‚0.9500691890716553 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.008240061812102795 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5383291244506836 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5570988059043884 particle/accuracy_7_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27083128690719604 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9718751311302185 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41862747073173523 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19257763028144836 particle/accuracy_3_of_5‚1 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 trainer/global_step‚4835 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1930348426103592 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3167998790740967 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5048816204071045 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872116446495056 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01684636063873768 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5266666412353516 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4399999976158142 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3819393515586853 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2524012625217438 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893453121185303 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.04605263099074364 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.253917396068573 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4282282292842865 jet/accuracy_5_of_6‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5806382894515991 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5136129856109619 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4867294132709503 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.981380820274353 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 ,&Purity/1FRt3SRR_¤rdqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4890873432159424 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9888585805892944 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_7‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17313380539417267 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0491071417927742 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5158131122589111 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18096168339252472 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.003885003738105297 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5348870158195496 +jet/accuracy_3_of_6‚0.13461539149284363 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.489358127117157 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16710127890110016 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9105955958366394 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3166666626930237 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14333891868591309 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4866046905517578 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4307992160320282 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.3045634627342224 particle/accuracy_3_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5489693880081177 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1084568127989769 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2117253690958023 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3501691222190857 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985421359539032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9903591871261597 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.040697675198316574 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_4_of_7‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12582726776599884 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333730697632 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12381265312433243 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6758420467376709 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.48796218633651733 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15103285014629364 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4277370572090149 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333730697632 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5103907585144043 particle/accuracy_2_of_6‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3770833909511566 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5235807299613953 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.43201756477355957 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.50529545545578 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.987646758556366 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4244185984134674 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.048245616257190704 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25718772411346436 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 *jet/accuracy_1_of_5‚0.9539042711257935 _step‚395 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3866535425186157 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07941609621047974 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0909963846206665 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1497783660888672 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.42672422528266907 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13991934061050415 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.30721673369407654 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.502053439617157 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019797751680016518 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5108358263969421 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3975308835506439 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15265677869319916 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027697613462805748 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14128482341766357 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3769146203994751 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23661068081855774 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.40909090638160706 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996397852897644 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 particle/accuracy_4_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9875825643539429 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.13063062727451324 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5501105785369873 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007980599999427795 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11213992536067963 +jet/accuracy_2_of_4‚0.37375032901763916 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0381944440305233 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25229883193969727 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9615384340286255 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.818446934223175 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14185787737369537 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006307793781161308 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5311943292617798 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5370370745658875 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 % _timestamp‚1.7884086716240041e+09 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935297012329 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49649596214294434 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18042725324630737 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7563822865486145 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5287542343139648 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1388888955116272 +jet/accuracy_2_of_2‚0.11023399978876114 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.975933849811554 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5443287491798401 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19279244542121887 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03365623205900192 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02638889104127884 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 jet/accuracy_7_of_7‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23083078861236572 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11527929455041885 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.48218390345573425 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835902452468872 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12126265466213226 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20681758224964142 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25879135727882385 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1329544186592102 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.37962961196899414 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966010451316833 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3434736132621765 ,jet/accuracy_4_of_5‚0.012314388528466225 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2122679203748703 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07025649398565292 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.163640096783638 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11868686974048615 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5287542343139648 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_1_of_4‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010273857973515987 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988448441028595 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1397927850484848 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5128205418586731 *loss/val_total_loss‚44.519397509470295 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47702404856681824 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5554741024971008 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4404905438423157 jet/accuracy_3_of_7‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.030480293557047844 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5767856240272522 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5788212418556213 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9895471930503845 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444465637207 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5776113867759705 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.47123652696609497 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_2_of_7‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5706093907356262 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6112637519836426 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35902532935142517 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5109378695487976 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9903367757797241 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20883190631866455 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 particle/accuracy_2_of_3‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4932035505771637 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0031256950460374355 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6446866393089294 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5314726233482361 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3095880448818207 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25519314408302307 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.510185182094574 particle/accuracy_5_of_5‚1 (particle/accuracy‚0.8512938022613525 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992858171463013 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19045399129390717 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.375520795583725 *particle/sensitivity‚0.317513108253479 particle/accuracy_5_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15526317059993744 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.977624237537384 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9822160005569458 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5646551847457886 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9858328700065613 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43061742186546326 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5209197402000427 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09899166971445084 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1537339687347412 +jet/accuracy_2_of_3‚0.24814863502979279 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 _runtime‚3774.613485205 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.403409868478775 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5027586221694946 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3545990586280823 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.023065801709890366 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1741379350423813 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5299615263938904 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14554636180400848 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5079470276832581 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19424891471862793 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.44730639457702637 ,jet/accuracy_3_of_3‚0.020286226645112038 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6107639074325562 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27975335717201233 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942994713783264 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219ÕhJ㎩§ =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20681758224964142 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25879135727882385 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1329544186592102 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.37962961196899414 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966010451316833 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3434736132621765 ,jet/accuracy_4_of_5‚0.012314388528466225 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2122679203748703 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07025649398565292 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.163640096783638 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11868686974048615 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5287542343139648 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_1_of_4‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010273857973515987 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988448441028595 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1397927850484848 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5128205418586731 *loss/val_total_loss‚44.519397509470295 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47702404856681824 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5554741024971008 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4404905438423157 jet/accuracy_3_of_7‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.030480293557047844 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5767856240272522 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5788212418556213 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9895471930503845 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444465637207 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5776113867759705 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.47123652696609497 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_2_of_7‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5706093907356262 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6112637519836426 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35902532935142517 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5109378695487976 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9903367757797241 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20883190631866455 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 particle/accuracy_2_of_3‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4932035505771637 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0031256950460374355 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6446866393089294 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5314726233482361 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3095880448818207 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25519314408302307 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.510185182094574 particle/accuracy_5_of_5‚1 (particle/accuracy‚0.8512938022613525 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992858171463013 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19045399129390717 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.375520795583725 *particle/sensitivity‚0.317513108253479 particle/accuracy_5_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15526317059993744 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.977624237537384 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FêrÍwRt*SRqqt2FBt/FBt_purity‚0.9822160005569458 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5646551847457886 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9858328700065613 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43061742186546326 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5209197402000427 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09899166971445084 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1537339687347412 +jet/accuracy_2_of_3‚0.24814863502979279 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 _runtime‚3774.613485205 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.403409868478775 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5027586221694946 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3545990586280823 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.023065801709890366 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1741379350423813 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5299615263938904 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14554636180400848 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5079470276832581 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19424891471862793 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.44730639457702637 ,jet/accuracy_3_of_3‚0.020286226645112038 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6107639074325562 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27975335717201233 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942994713783264 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.39981621503829956 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44902023673057556 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 particle/accuracy_4_of_5‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07758019864559174 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5384051203727722 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4583333432674408 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973047971725464 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016219450160861015 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.29907408356666565 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.38271602988243103 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24074071645736694 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 particle/accuracy_2_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.033965837210416794 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4095860421657562 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48938047885894775 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4799463152885437 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5107930302619934 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 epoch‚3 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 *jet/accuracy_1_of_3‚0.8627812266349792 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.49161630868911743 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019806981086730957 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 *jet/accuracy_3_of_4‚0.0641067773103714 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.163640096783638 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4951401948928833 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9983371496200562 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_4‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5388551354408264 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666567325592 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14190487563610077 particle/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5336984395980835 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9905575513839722 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1450464129447937 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4289851486682892 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4889158010482788 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03929402306675911 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09259707480669022 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9773243069648743 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3023809790611267 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4303557872772217 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.191361203789711 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 +particle/specificity‚0.9500691890716553 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.008240061812102795 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5383291244506836 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5570988059043884 particle/accuracy_7_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27083128690719604 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9718751311302185 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41862747073173523 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19257763028144836 particle/accuracy_3_of_5‚1 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 trainer/global_step‚4835 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1930348426103592 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3167998790740967 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5048816204071045 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872116446495056 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01684636063873768 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5266666412353516 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4399999976158142 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3819393515586853 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2524012625217438 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893453121185303 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.04605263099074364 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.253917396068573 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4282282292842865 jet/accuracy_5_of_6‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5806382894515991 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5136129856109619 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4867294132709503 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.981380820274353 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4890873432159424 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9888585805892944 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_7‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17313380539417267 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0491071417927742 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5158131122589111 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18096168339252472 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.003885003738105297 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5348870158195496 +jet/accuracy_3_of_6‚0.13461539149284363 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.489358127117157 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16710127890110016 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9105955958366394 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3166666626930237 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14333891868591309 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4866046905517578 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4307992160320282 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.3045634627342224 particle/accuracy_3_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5489693880081177 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1084568127989769 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2117253690958023 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3501691222190857 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985421359539032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9903591871261597 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.040697675198316574 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_4_of_7‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12582726776599884 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333730697632 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12381265312433243 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6758420467376709 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.48796218633651733 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15103285014629364 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4277370572090149 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333730697632 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5103907585144043 particle/accuracy_2_of_6‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3770833909511566 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5235807299613953 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.43201756477355957 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.50529545545578 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.987646758556366 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4244185984134674 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.048245616257190704 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25718772411346436 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 *jet/accuracy_1_of_5‚0.9539042711257935 _step‚395 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3866535425186157 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07941609621047974 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0909963846206665 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1497783660888672 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.42672422528266907 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13991934061050415 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.30721673369407654 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.502053439617157 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019797751680016518 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5108358263969421 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3975308835506439 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15265677869319916 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027697613462805748 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14128482341766357 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3769146203994751 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23661068081855774 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.40909090638160706 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996397852897644 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 particle/accuracy_4_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9875825643539429 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.13063062727451324 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5501105785369873 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007980599999427795 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11213992536067963 +jet/accuracy_2_of_4‚0.37375032901763916 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0381944440305233 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25229883193969727 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9615384340286255 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.818446934223175 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14185787737369537 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006307793781161308 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5311943292617798 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5370370745658875 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 % _timestamp‚1.7884086716240041e+09 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935297012329 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49649596214294434 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18042725324630737 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7563822865486145 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5287542343139648 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1388888955116272 +jet/accuracy_2_of_2‚0.11023399978876114 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.975933849811554 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5443287491798401 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19279244542121887 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03365623205900192 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02638889104127884 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 jet/accuracy_7_of_7‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23083078861236572 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11527929455041885 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.48218390345573425 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835902452468872 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12126265466213226‹R¯m^‘‚(:† åæãÔͦ¦ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.095' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.819 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mÉ‘DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91689348 trainer/global_step‚4849 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279247217826644 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚397 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4068108491776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282610318156288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4052473831252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23297226 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286677752282624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4053128269008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282610318156288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4266198818816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64556286 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286678426648064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70708310 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308571393884672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4266035194880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282610300153344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70708308 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41259508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279247217826644 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚397 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4068108491776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282610318156288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4052473831252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23297226 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286677752282624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4053128269008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282610318156288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4266198818816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64556286 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286678426648064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70708310 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308571393884672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4266035194880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282610300153344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70708308 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41259508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0|QPu2—- _wandbruntime‚3789  _runtime‚3789ÒÚÿæ&˜àM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312900126533120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64568953 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70722326 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282666653192192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282666671195136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91687764 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4053944704180 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4068927932928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275249665165312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4849 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91707828 % _timestamp‚1.7884086863256397e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286735599128064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4068271557120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286734924749312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282666671195136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41267678 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279303609869492 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64569395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4068271557120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324‹À“5 ™åM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282666653192192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282666671195136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91687764 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4053944704180 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4068927932928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2752496651653CNo¶12 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4849 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91707828 % _timestamp‚1.7884086863256397e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286735599128064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4068271557120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286734924749312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282666671195136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41267678 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279303609869492 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64569395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4068271557120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312900126533120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64568953 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70722326 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844Žþ¥ 2š- _wandbruntime‚3789  _runtime‚3789ãÍLxZ›Ô _runtime‚ 3789.31481999 2loss/FBt3/assignment_loss‚0.052069372774932496 epoch‚4 2loss/SRqqt1/assignment_loss‚1.9883508412488575 0loss/FBt1/assignment_loss‚1.0121746212244034 0loss/SRqqt1/detection_loss‚2.604358655625376 1loss/SRqqt2/detection_loss‚1.4763689033560834 trainer/global_step‚4849 0loss/FRt1/assignment_loss‚3.8555194464596836 2loss/SRqqt3/assignment_loss‚1.1840817670095456 0loss/FBt4/assignment_loss‚0.4361574047555526 .loss/FRt3/detection_loss‚1.551957372713972 0loss/SRqqt4/detection_loss‚1.898279252297738 /loss/FBt1/detection_loss‚3.2407976388931274 /loss/FBt3/detection_loss‚0.4898917244998022 %loss/total_loss‚45.86597131844365 /loss/FRt2/detection_loss‚2.2498575741494142 _step‚399 0loss/FRt2/assignment_loss‚2.8524505341494524 0loss/FRt4/assignment_loss‚2.6749569856843283 1loss/SRqqt3/detection_loss‚1.1123955289590848 /loss/FBt4/detection_loss‚3.5969927509625754 % _timestamp‚1.7884086863274412e+09 /loss/FRt3/assignment_loss‚1.594253522378427 2loss/SRqqt2/assignment_loss‚1.7012609769037719 2loss/SRqqt4/assignment_loss‚1.4734327246745427 0loss/FBt2/assignment_loss‚0.2615702827461064 .loss/FRt1/detection_loss‚2.850632748820565 .loss/FRt4/detection_loss‚2.454815931791483 /loss/FBt2/detection_loss‚3.2533447563648226Öý_œÙ 1loss/SRqqt3/detection_loss‚1.1123955289590848 /loss/FBt4/detection_loss‚3.5969927509625754 % _timestamp‚1.7884086863274412e+09 /loss/FRt3/assignment_loss‚1.594253522378427 2loss/SRqqt2/assignment_loss‚1.7012609769037719 2loss/SRqqt4/assignment_loss‚1.4734327246745427 0loss/FBt2/assignment_loss‚0.2615702827461064 .loss/FRt1/detection_loss‚2.850632748820565 .loss/FRt4/detection_loss‚2.454815931791483 /loss/FBt2/detection_loss‚3.2533447563648226 _runtime‚ 3789.31481999 2loss/FBt3/assignment_loss‚0.052069372774932496 epoch‚4 2loss/SRqqt1/assignment_loss‚1.9883508412488575 0loss/FBt1/assignment_loss‚1.0121746212244034 0loss/SRqqt1/detection_loss‚2.604358655625376 1loss/SRqqt2/detection_loss‚1.4763689033560834 trainer/global_step‚4849 0loss/FRt1/assignment_loss‚3.8555194464596836 2loss/SRqqt3/assignment_loss‚1.1840817670095456 0loss/FBt4/assignment_loss‚0.4361574047555526 .loss/FRt3/detection_loss‚1.551957372713972 0loss/SRqqt4/detection_loss‚1.898279252297738 /loss/FBt1/detection_loss‚3.2407976388931274 /loss/FBt3/detection_loss‚0.4898917244998022 %loss/total_loss‚45.86597131844365 /loss/FRt2/detection_loss‚2.2498575741494142 _step‚399 0loss/FRt2/assignment_loss‚2.8524505341494524 0loss/FRt4/assignment_loss‚2.6749569856843283î8k‚(:… ôæãÔµœ˜ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚132.38' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`§>8až‚(:Ö ôæãÔ†ç˜  disk.sda.in‚0 disk.sda3.out‚ 192.29296875 cpu‚25.367200715719225 disk.sda1.out‚0 disk.rbd1.out‚ 18.6826171875& disk.nvme0n1.out‚16324.18408203125 disk.dm-1.out‚ 192.29296875 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 29.234375# disk.dm-0.out‚16324.18408203125 disk.sda.out‚ 192.29296875 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 29.234375* disk./.usagePercent‚15.748596341410664 disk.dm-0.in‚ 29.234375 disk.sda2.out‚0 network.recv‚ 1.420348e+06 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1425.33984375) proc.memory.percent‚17.10193157196045 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0% disk./.usageGB‚234.66546630859375 disk.drbd1716.in‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚16324.18408203125$ memory_percent‚91.30041599273682# proc.memory.rssMB‚ 2801.98046875 proc.cpu.threads‚17 network.sent‚ 1.1130061e+07{…â‘Ÿ‚(:† ƒçãÔÈÎô gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚157.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.427 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#cÊBY ‚(:Î ƒçãÔñä¶  disk.nvme0c0n1.in‚ 29.234375* disk./.usagePercent‚15.860247627281213 disk.sda.out‚ 194.77734375 disk.drbd2266.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚16806.802734375 disk.rbd1.out‚ 18.71484375) proc.memory.availableMB‚ 1425.33984375 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0$ disk.nvme0n1.out‚16806.802734375! disk.dm-0.out‚16806.802734375 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 network.recv‚ 1.426698e+06% disk./.usageGB‚236.32915115356445 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.drbd2080.out‚0$ memory_percent‚91.30041599273682# proc.memory.rssMB‚ 2801.98046875 disk.nvme0n1.in‚ 29.234375 disk.dm-1.out‚ 194.77734375 disk.drbd1716.in‚0 cpu‚25.365956774515737 disk.dm-0.in‚ 29.234375 disk.dm-1.in‚0 disk.drbd2080.in‚0) proc.memory.percent‚17.10193157196045 network.sent‚ 1.1238477e+07 disk.sda.in‚0 disk.sda3.out‚ 194.77734375 disk.sda1.out‚0 disk.sda2.out‚0ù=Â’¡‚(:‡ ’çãÔÞß– gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dÕî2¢- _wandbruntime‚3825  _runtime‚3825±“JŸÇ£Á & lr-AdamW/pg1‚0.0002041165838264926 & lr-AdamW/pg2‚0.0002041165838264926 trainer/global_step‚4899 % _timestamp‚1.7884087220192494e+09 _runtime‚3825.006661662 _step‚400N| ¥Ì¤Æ _runtime‚3825.006661662 _step‚400 & lr-AdamW/pg1‚0.0002041165838264926 & lr-AdamW/pg2‚0.0002041165838264926 trainer/global_step‚4899 % _timestamp‚1.7884087220192494e+09᥿ô2¥- _wandbruntime‚3825  _runtime‚3825*®Ý'¦×O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277939950523392 _runtime‚3825.007931438 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92595362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285427970098688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277939967562752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311653930767872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164004470 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4899 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4109094297600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92615400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289536390030848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41667910 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282033259694052 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315963254932992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109094297600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289537064396288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4108437935104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23523549 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289536390030848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71409108 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4093309170660 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65191030 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4309300225024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164024510 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92595362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65191459 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚285427952095744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41667908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311653954707968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4108437935104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71409108 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285427970098688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23523122 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315963067368960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4309136601088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289536390030848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41667910 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282033259694052 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315963254932992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109094297600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289537064396288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4108437935104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23523549 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289536390030848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71409108 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4093309170660 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65191030 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4309300225024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164024510 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92595362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65191459 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚285427952095744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41667908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311653954707968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4108437935104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71409108 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285427970098688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23523122 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315963067368960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4309136601088 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128‘W T¨‚(:É ’çãÔ×Ô‚  disk.dm-1.in‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚17294.599609375# proc.memory.rssMB‚ 2801.98046875* disk./.usagePercent‚15.932137614986297 disk.drbd2265.out‚0 disk.sda.out‚ 195.4765625 disk.sda2.in‚0# memory_percent‚91.3020372390747% disk./.usageGB‚237.40036392211914 disk.dm-1.out‚ 195.4765625 disk.drbd2265.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 18.98046875 network.sent‚ 1.124432e+07 network.recv‚ 1.431278e+06! disk.dm-0.out‚17294.724609375 disk.sda3.out‚ 195.4765625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 29.234375 disk.nvme0n1.in‚ 29.234375$ disk.nvme0n1.out‚17294.599609375) proc.memory.availableMB‚ 1425.07421875) proc.memory.percent‚17.10193157196045 disk.sda3.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 cpu‚25.364731308065405 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 29.234375p~«2©- _wandbruntime‚3825  _runtime‚3825¦["‡æ&ªàM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282089651736900 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289594236876288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285484305134592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4109913738752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚285484323137536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4109913738752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109257362944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164036940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41676078 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚402 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289593562497536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71423126 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92633880 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23527980 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285484323137536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 $ _timestamp‚1.788408722703027e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164057006 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92613816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289594236876288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65203621 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23527545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282088980259576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71423124 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4109257362944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277994854653952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277994871693312 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚4899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71423124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310158338560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚3825.690961927 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4309992630784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311715581505536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4094780043588 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521¾W;&ë&«åM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310158338560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚3825.690961927 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4309992630784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311715581505536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4094780043588 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282089651736900 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289594236876288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285484305134592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4109913738752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚285484323137536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4109913738752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109257362944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164036940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41676078 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚402 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289593562497536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71423126 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92633880 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23527980 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285484323137536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 $ _timestamp‚1.788408722703027e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164057006 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92613816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289594236876288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65203621 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23527545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282088980259576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71423124 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4109257362944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277994854653952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277994871693312 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚4899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71423124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128’ìçÚ2¬- _wandbruntime‚3825  _runtime‚3825ÛIÐeW­Ñ 0loss/FRt1/assignment_loss‚2.7996268639197717 0loss/FBt4/assignment_loss‚0.3068230098520871 /loss/FRt2/detection_loss‚2.3314969866740993 0loss/SRqqt2/detection_loss‚1.791363952251581 _runtime‚3825.691895479 2loss/SRqqt1/assignment_loss‚1.5674735618755222 1loss/SRqqt3/detection_loss‚0.8275077823135588 ,loss/FBt1/detection_loss‚2.8325035572052 /loss/FRt2/assignment_loss‚3.265535222321022 0loss/FRt3/assignment_loss‚1.8889676567755247 0loss/SRqqt1/detection_loss‚2.149863551557064 2loss/SRqqt3/assignment_loss‚1.2274102740173722 .loss/FRt1/detection_loss‚2.993507518218114 /loss/FRt4/detection_loss‚2.1907517427489873 %loss/total_loss‚42.09399921034724 _step‚403 2loss/SRqqt4/assignment_loss‚1.1466866109520197 1loss/FBt3/assignment_loss‚0.15886906802762119 .loss/FBt4/detection_loss‚3.255069488286972 epoch‚4 0loss/FRt4/assignment_loss‚2.7755903814520155 2loss/SRqqt2/assignment_loss‚1.5318481177091599 /loss/FRt3/detection_loss‚1.9226184726545685 % _timestamp‚1.7884087227045493e+09 1loss/FBt1/assignment_loss‚0.15105497688255126 1loss/SRqqt4/detection_loss‚1.8516143760085106 .loss/FBt2/detection_loss‚2.273539651523937 0loss/FBt2/assignment_loss‚0.3830196429890665 .loss/FBt3/detection_loss‚0.471256744130917 trainer/global_step‚4899Ã5‡\®Ö /loss/FRt2/assignment_loss‚3.265535222321022 0loss/FRt3/assignment_loss‚1.8889676567755247 0loss/SRqqt1/detection_loss‚2.149863551557064 2loss/SRqqt3/assignment_loss‚1.2274102740173722 .loss/FRt1/detection_loss‚2.993507518218114 /loss/FRt4/detection_loss‚2.1907517427489873 %loss/total_loss‚42.09399921034724 _step‚403 2loss/SRqqt4/assignment_loss‚1.1466866109520197 1loss/FBt3/assignment_loss‚0.15886906802762119 .loss/FBt4/detection_loss‚3.255069488286972 epoch‚4 0loss/FRt4/assignment_loss‚2.7755903814520155 2loss/SRqqt2/assignment_loss‚1.5318481177091599 /loss/FRt3/detection_loss‚1.9226184726545685 % _timestamp‚1.7884087227045493e+09 1loss/FBt1/assignment_loss‚0.15105497688255126 1loss/SRqqt4/detection_loss‚1.8516143760085106 .loss/FBt2/detection_loss‚2.273539651523937 0loss/FBt2/assignment_loss‚0.3830196429890665 .loss/FBt3/detection_loss‚0.471256744130917 trainer/global_step‚4899 0loss/FRt1/assignment_loss‚2.7996268639197717 0loss/FBt4/assignment_loss‚0.3068230098520871 /loss/FRt2/detection_loss‚2.3314969866740993 0loss/SRqqt2/detection_loss‚1.791363952251581 _runtime‚3825.691895479 2loss/SRqqt1/assignment_loss‚1.5674735618755222 1loss/SRqqt3/detection_loss‚0.8275077823135588 ,loss/FBt1/detection_loss‚2.8325035572052“b§þ‘¯‚(:† ¡çãÔÓ•‰ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.715' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.343 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}7^Q°‚(:Æ ¡çãÔԽР disk.drbd2265.out‚0 disk.sda.in‚0 cpu‚25.363385767591705 disk.nvme0n1.in‚ 29.234375( disk.nvme0c0n1.out‚19209.21630859375 network.recv‚ 1.435545e+06 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 network.sent‚ 1.1250593e+07 disk.rbd1.in‚0 disk.dm-0.in‚ 29.234375 disk.dm-1.in‚0 disk.dm-1.out‚196.1875* disk./.usagePercent‚16.049148674593695& disk.nvme0n1.out‚19209.21630859375# disk.dm-0.out‚19209.21630859375% disk./.usageGB‚239.14391326904297 disk.sda3.out‚196.1875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚ 29.234375 disk.drbd1716.out‚0 disk.rbd1.out‚ 19.04296875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda.out‚196.1875$ memory_percent‚91.28351211547852# proc.memory.rssMB‚ 2801.98046875' proc.memory.availableMB‚ 1428.109375) proc.memory.percent‚17.10193157196045 proc.cpu.threads‚17·’±‚(:‡ °çãÔèó€ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.981' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7962 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#÷»'V²‚(:Ë °çãÔDZŸ  disk.dm-1.out‚ 196.640625 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2801.98046875 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 29.234375 disk.drbd1716.in‚0 disk.sda.out‚ 196.640625* disk./.usagePercent‚16.126867185830832 disk.dm-0.in‚ 29.234375 disk.sda.in‚0 disk.sda3.out‚ 196.640625 disk.rbd1.in‚0& disk.nvme0n1.out‚21788.61083984375$ memory_percent‚94.17939186096191 network.recv‚ 1.442449e+06 disk.nvme0n1.in‚ 29.234375( disk.nvme0c0n1.out‚21788.61083984375 disk.drbd1716.out‚0 disk.sda1.out‚0 cpu‚25.36215242457773 disk.rbd0.in‚0 network.sent‚ 1.1358515e+07# disk.dm-0.out‚21788.61083984375 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 19.04296875' proc.memory.availableMB‚ 953.6484375 proc.cpu.threads‚17 disk.dm-1.in‚0) proc.memory.percent‚17.10193157196045% disk./.usageGB‚240.30197525024414 disk.drbd2265.in‚0‹y'Æ2³- _wandbruntime‚3861  _runtime‚3861‰ÌU-É´Ã % _timestamp‚1.7884087585829263e+09 _runtime‚3861.570385523 _step‚404 ' lr-AdamW/pg1‚0.00020237253171493882 ' lr-AdamW/pg2‚0.00020237253171493882 trainer/global_step‚4949~%[‹ÎµÈ ' lr-AdamW/pg2‚0.00020237253171493882 trainer/global_step‚4949 % _timestamp‚1.7884087585829263e+09 _runtime‚3861.570385523 _step‚404 ' lr-AdamW/pg1‚0.00020237253171493882”àŒ2¶- _wandbruntime‚3861  _runtime‚3861md¬![ ·ÔO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72109910 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42076308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23750882 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292395702144512 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93541452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319088684844032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚ 3861.5713274 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292395702144512 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292395027779072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149423740928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284819301561460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72109908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4149423740928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4134144510068 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288245604038144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚288245604038144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319088866116608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314736491591168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280685157051392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 trainer/global_step‚4949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165631322 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93521414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65826755 $ _timestamp‚1.788408758583662e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65827192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23750447 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284819973038576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚405 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72109908 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4150080103424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280685174090752‰¥Ó¥ß'¸ÙO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292395702144512 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292395027779072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149423740928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284819301561460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72109908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4149423740928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4134144510068 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288245604038144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚288245604038144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319088866116608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314736491591168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280685157051392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 trainer/global_step‚4949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165631322 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93521414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65826755 $ _timestamp‚1.788408758583662e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65827192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23750447 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284819973038576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚405 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72109908 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4150080103424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280685174090752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72109910 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42076308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23750882 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292395702144512 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93541452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319088684844032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚ 3861.5713274•‚P¡”2¹- _wandbruntime‚3862  _runtime‚3862&âsç&ºáM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288301957076992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4353069066240 _runtime‚3862.253156154 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165663792 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65839489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4134960945032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23755013 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165683858 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288301957076992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292452874624512 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4150899544576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4353236871168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42084478 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165663792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292452874624512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288301975079936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319151379200000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319151187454976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284875022126984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4150243168768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292452200245760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72123924 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280740061181952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23755450 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314798142328832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280740078221312 _step‚406 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42084476 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72123926 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884087592650225e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314798118388736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4135615382996 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284875693604308 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚4949 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72123924 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314798118388736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4135615382996 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284875693604308 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚4949 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72123924 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65839489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4134960945032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23755013 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165683858 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288301957076992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292452874624512 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4150899544576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4353236871168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42084478 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165663792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292452874624512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/se´Š4‚†gment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288301975079936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319151379200000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319151187454976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284875022126984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4150243168768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292452200245760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72123924 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280740061181952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23755450 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314798142328832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280740078221312 _step‚406 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42084476 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72123926 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884087592650225e+09 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23979068 % _timestamp‚1.7884087947720187e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295253665527296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4190409546752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42484708 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317819028474368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66463329 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72810708 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72810708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66463778 trainer/global_step‚4999 _runtime‚ 3897.75967686 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4174979849476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94467504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291063255980544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025°PI¼á'ÈÛO % _timestamp‚1.7884087947720187e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295253665527296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4190409546752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42484708 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317819028474368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66463329 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72810708 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72810708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66463778 trainer/global_step‚4999 _runtime‚ 3897.75967686 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4174979849476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94467504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291063255980544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4191065909248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94447466 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322214395778048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291063255980544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4175634287232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23979068™²“1•2É- _wandbruntime‚3898  _runtime‚3898×—ÛTÊàM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3898.445023523 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72824726 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72824724 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4175796284440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚4999 $ _timestamp‚1.788408795457042e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42492876 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 aODeviceStatsMonitor.on_train_batch_end/inactive_split_byte¤›Ã’!s.small_pool.allocated‚ 4396416798720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283485284749312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72824724 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66475945 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167290644 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4396257382400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317880679212032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291119609019392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66476404 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287661735471716 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291119609019392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42492878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94465920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295311512372736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322276912654336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4191885350400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291119627022336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3898.445023523 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72824726 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72824724 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4175796284440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚4999 $ _timestamp‚1.788408795457042e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42492876 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4396416798720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283485284749312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72824724 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66475945 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167290644 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4396257382400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317880679212032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291119609019392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66476404 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287661735471716 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291119609019392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42492878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94465920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295311512372736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322276912654336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4191885350400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291119627022336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67100400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293880925925888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293880925925888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320901565357568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24206837 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298112303275520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4438725180928 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286175570107392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298112977640960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325340290538496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290391385296276 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4232051715072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67099945 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286175587146752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95393556 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73511508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42893110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95373518 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872¯W¹à'ÚÚO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4232051715072 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298112303275520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4438725180928 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286175570107392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298112977640960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325340290538496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290391385296276 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4232051715072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67099945 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286175587146752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95393556 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73511508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42893110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95373518 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73511508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168885026 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4438557362688 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24207290 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290392056773392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4216469626640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298112303275520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168905066 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚293880907922944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42893108 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4215815188884 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚320901541417472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884088312058089e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5049 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67100400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293880925925888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293880925925888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320901565357568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24206837 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884088318881018e+09 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67112694 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73525526 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42901276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168917496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95391972 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232214780416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24211418 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73525524 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4439581667328 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298170150120960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293937260961792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73525524 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320963192155136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293937260961792 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298170150120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚325402612308992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ¥/?f!ÝæM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚5049 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4232871156224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73525526 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95391972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4217286061812 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4216631623848 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42901276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168917496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95391972 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232214780416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24211418 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73525524 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4439581667328 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298170150120960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293937260961792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73525524 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320963192155136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293937260961792 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298170150120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚325402612308992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4232214780416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4232871156224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286230491277312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884088318881018e+09 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚‚b¸†0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67112694 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73525526 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300970941023744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323984078300672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296698559865344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4481546299904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚293177427163684 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4256650528292 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293178098640800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67736676 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67736226 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96319608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96319608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300971615389184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296698559865344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170531918 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24434718 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74212310 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96299570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚417 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96299570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328465624600576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4273037520896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74212308 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300970941023744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4273037520896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 trainer/global_step‚5099 % _timestamp‚1.7884088673869114e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24435166 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4272381158400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4257304966048 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170511878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323984102240768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 6544377567¥â'êÜO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚417 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96299570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328465624600576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4273037520896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74212308 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300970941023744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4273037520896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 trainer/global_step‚5099 % _timestamp‚1.7884088673869114e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24435166 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4272381158400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4257304966048 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170511878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323984102240768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296698577868288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296698577868288 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300970941023744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323984078300672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296698559865344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4481546299904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚293177427163684 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4256650528292 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293178098640800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67736676 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67736226 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96319608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96319608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300971615389184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296698559865344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170531918 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24434718 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74212310 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96299570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605¡÷8`2ë- _wandbruntime‚3971  _runtime‚3971s WcìáM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67748901 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚5099 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296754912904192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74226326 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96318024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170544348 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4273856962048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67749349 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301028787869184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296754912904192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _runtime‚3971.063940394 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96338088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24439671 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324045729038336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4273856962048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296754930907136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_enÄõH„d/segment.all.peak‚844 % _timestamp‚1.7884088680759153e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43309678 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4257466963256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4482578179072 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170564414 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96318024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328528151995392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4482422957056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296754930907136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96338088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43309676 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324045752978432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430ÐÔ÷»ì&íæM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301028787869184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296754912904192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _runtime‚3971.063940394 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96338088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24439671 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324045729038336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4273856962048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296754930907136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884088680759153e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43309678 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4257466963256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4482578179072 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170564414 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96318024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328528151995392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4482422957056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296754930907136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96338088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43309676 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324045752978432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67748901 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚5099 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296754912904192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74226326 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96318024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170544348 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4273856962048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67749349 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414¢k–“¿2î- _wandbruntime‚3971  _runtime‚3971~›ÇWïÑ /loss/FRt4/assignment_loss‚2.804703691005707 1loss/SRqqt4/detection_loss‚1.8513758362464185 1loss/SRqqt2/assignment_loss‚1.681663052147349 trainer/global_step‚5099 _runtime‚ 3971.06506786 /loss/FRt2/assignment_loss‚3.420143367101749 1loss/SRqqt1/assignment_loss‚1.939325185821337 /loss/FRt3/detection_loss‚1.6958467517654563 /loss/FRt4/detection_loss‚2.2208858126401902 0loss/FRt3/assignment_loss‚1.3776706027430157 2loss/SRqqt3/assignment_loss‚0.9737947116536816 1loss/SRqqt4/assignment_loss‚1.675491583516013 0loss/FBt1/assignment_loss‚0.3153158384928929 /loss/FRt1/assignment_loss‚3.603934269685012 1loss/SRqqt2/detection_loss‚1.9408017669693898 1loss/SRqqt3/detection_loss‚0.9043760483618826 /loss/FBt1/detection_loss‚2.5261906584103904 /loss/FBt4/detection_loss‚3.2251981496810913 epoch‚4 1loss/FBt2/assignment_loss‚0.07752255173399902 1loss/FBt4/assignment_loss‚0.04364404461812228 .loss/FBt2/detection_loss‚3.484615159034729 1loss/FBt3/assignment_loss‚0.12302286274630103 /loss/FRt2/detection_loss‚2.2611609684924283 %loss/total_loss‚44.11755985174675 .loss/FRt1/detection_loss‚3.214546960133773 0loss/SRqqt1/detection_loss‚2.255589022355921 .loss/FBt3/detection_loss‚0.500740956389896 % _timestamp‚1.7884088680776691e+09 _step‚419†A\ðÖ 1loss/SRqqt2/assignment_loss‚1.681663052147349 trainer/global_step‚5099 _runtime‚ 3971.06506786 /loss/FRt2/assignment_loss‚3.420143367101749 1loss/SRqqt1/assignment_loss‚1.939325185821337 /loss/FRt3/detection_loss‚1.6958467517654563 /loss/FRt4/detection_loss‚2.2208858126401902 0loss/FRt3/assignment_loss‚1.3776706027430157 2loss/SRqqt3/assignment_loss‚0.9737947116536816 1loss/SRqqt4/assignment_loss‚1.675491583516013 0loss/FBt1/assignment_loss‚0.3153158384928929 /loss/FRt1/assignment_loss‚3.603934269685012 1loss/SRqqt2/detection_loss‚1.9408017669693898 1loss/SRqqt3/detection_loss‚0.9043760483618826 /loss/FBt1/detection_loss‚2.5261906584103904 /loss/FBt4/detection_loss‚3.2251981496810913 epoch‚4 1loss/FBt2/assignment_loss‚0.07752255173399902 1loss/FBt4/assignment_loss‚0.04364404461812228 .loss/FBt2/detection_loss‚3.484615159034729 1loss/FBt3/assignment_loss‚0.12302286274630103 /loss/FRt2/detection_loss‚2.2611609684924283 %loss/total_loss‚44.11755985174675 .loss/FRt1/detection_loss‚3.214546960133773 0loss/SRqqt1/detection_loss‚2.255589022355921 .loss/FBt3/detection_loss‚0.500740956389896 % _timestamp‚1.7884088680776691e+09 _step‚419 /loss/FRt4/assignment_loss‚2.804703691005707 1loss/SRqqt4/detection_loss‚1.8513758362464185£­˜¥‘ñ‚(:’ ¨èãÔ•æ² gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.463' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.492599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€(µTò‚(:É ¨èãÔá«´  disk.drbd1716.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 202.734375* disk./.usagePercent‚16.635516523258755 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 29.28515625 disk.dm-0.out‚34900.41015625 disk.drbd2265.in‚0& proc.memory.availableMB‚ 953.203125 network.sent‚ 1.1713686e+07 disk.sda1.in‚0 disk.drbd2266.out‚0) proc.memory.percent‚17.10193157196045 cpu‚25.352660698370478 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 202.734375 disk.drbd2266.in‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd2265.out‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2801.98046875% disk.nvme0c0n1.out‚34900.41015625 disk.nvme0n1.in‚ 29.28515625$ memory_percent‚94.18210983276367 disk.rbd1.out‚ 19.505859375 disk.sda.in‚0 disk.sda3.out‚ 202.734375# disk.nvme0n1.out‚34900.41015625 disk.drbd2080.out‚0 disk.sda3.in‚0 network.recv‚ 1.484841e+06% disk./.usageGB‚247.88121795654297KA¯œó‚(:‘ ·èãÔåó gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.41' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.681999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I·Cäfô‚(:Û ·èãÔ‘ Ò  disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0 cpu‚25.351490830250295 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 953.05859375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 203.27734375 disk.drbd2265.in‚0$ memory_percent‚94.18299198150635) proc.memory.percent‚17.10193157196045 network.recv‚ 1.492126e+06 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.rbd1.out‚ 19.5380859375 disk.nvme0n1.in‚ 29.28515625 disk.dm-0.in‚ 29.28515625 disk.sda.out‚ 203.27734375 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚34994.11572265625 disk.dm-1.out‚ 203.27734375 network.sent‚ 1.1822549e+07* disk./.usagePercent‚16.632017156273225% disk./.usageGB‚247.82907485961914 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2801.98046875 disk.sda2.out‚0( disk.nvme0c0n1.out‚34994.11572265625& disk.nvme0n1.out‚34994.11572265625sã`»õ‚(:… ÆèãÔÄŸä gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V.'%bö‚(:× ÆèãÔ¨½ÿ  network.recv‚ 1.495755e+06( disk./.usagePercent‚16.6321036868198 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 29.28515625 disk.sda2.in‚0# memory_percent‚94.1831111907959 disk.dm-1.out‚ 203.80859375 disk.sda3.out‚ 203.80859375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd1716.in‚0# disk.dm-0.out‚34994.73291015625) proc.memory.percent‚17.10193157196045 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 19.7177734375 disk.sda.in‚0 network.sent‚ 1.1827438e+07 disk.drbd2266.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚34994.73291015625# proc.memory.rssMB‚ 2801.98046875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚34994.73291015625 disk.drbd2265.in‚0' proc.memory.availableMB‚ 953.0390625 cpu‚25.350363979067115% disk./.usageGB‚247.83036422729492 disk.dm-0.in‚ 29.28515625 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 203.80859375ìòÁ2÷- _wandbruntime‚4006  _runtime‚4006Wƒ4wÈø trainer/global_step‚5149 $ _timestamp‚1.788408903730181e+09 _runtime‚4006.717630007 _step‚420 ' lr-AdamW/pg1‚0.00019526486777624656 ' lr-AdamW/pg2‚0.00019526486777624656DºVùÍùÇ _runtime‚4006.717630007 _step‚420 ' lr-AdamW/pg1‚0.00019526486777624656 ' lr-AdamW/pg2‚0.00019526486777624656 trainer/global_step‚5149 $ _timestamp‚1.788408903730181e+09¤nd2ú- _wandbruntime‚4006  _runtime‚4006!¿‚‚]$ûÖO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303829578771968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97225622 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68373156 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327066615183872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303830253137408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299516211807744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303829578771968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299516229810688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299516229810688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299516211807744 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97245660 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24663679 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303830253137408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _step‚421 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4314023326720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291666000202752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4314023326720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97225622 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4006.718645922 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172158770 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74913108 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4313366964224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24663248 trainer/global_step‚5149 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43709908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331591226041856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295964140508208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4298140305456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172138730 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4313366964224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74913108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327066639123968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295963469031092 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788408903730956e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74913110 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97245660 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4524610857984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.•µÆŠon_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68373589 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331591417800192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4524778676224 ª Úá'üÛO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4314023326720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97225622 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4006.718645922 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172158770 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74913108 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4313366964224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24663248 trainer/global_step‚5149 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43709908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331591226041856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295964140508208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4298140305456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172138730 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4313366964224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74913108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327066639123968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295963469031092 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788408903730956e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74913110 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97245660 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4524610857984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68373589 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331591417800192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4524778676224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303829578771968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97225622 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68373156 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327066615183872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303830253137408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299516211807744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303829578771968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299516229810688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299516229810688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299516211807744 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97245660 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24663679 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303830253137408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _step‚421 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4314023326720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291666000202752¥Ėl2ý- _wandbruntime‚4007  _runtime‚4007×OÜuç&þáM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43718076 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291720904333312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43718078 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303886751238656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4314186392064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4298956740628 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299572582849536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299572564846592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24667771 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97264140 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4314842767872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296019861073940 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚4007.400179666 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299572582849536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4525470192128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74927126 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97244076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97264140 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68385847 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97244076 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296019189596616 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚422 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74927124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68386291 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327128265921536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291720887293952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172171200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303887425617408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4525635899904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4298302302664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4314842767872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74927124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24668213 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884089044121532e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4314186392064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331653925761536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299572564846592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚5149 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0wR>ì&ÿæM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4525635899904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4298302302664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4314842767872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74927124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24668213 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884089044121532e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4314186392064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331653925761536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299572564846592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚5149 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43718076 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291720904333312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43718078 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303886751238656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4314186392064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4298956740628 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299572582849536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299572564846592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24667771 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97264140 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4314842767872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296019861073940 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚4007.400179666 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299572582849536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4525470192128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74927126 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97244076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97264140 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68385847 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97244076 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296019189596616 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚422 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74927124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68386291 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327128265921536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291720887293952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172171200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303887425617408¦ƒÇýé2€- _wandbruntime‚4007  _runtime‚4007C¹ÇåXÒ 0loss/FBt1/assignment_loss‚0.4926990928339695 0loss/SRqqt1/detection_loss‚2.302169478300846 2loss/SRqqt2/assignment_loss‚1.7469079666770995 /loss/FBt4/assignment_loss‚0.790176491314196 _step‚423 1loss/SRqqt4/assignment_loss‚1.435134909986969 0loss/FBt2/assignment_loss‚0.1999394478409613 1loss/SRqqt3/detection_loss‚1.0326587919201424 0loss/FRt4/assignment_loss‚3.3901491923765703 .loss/FRt2/detection_loss‚2.141174684350307 /loss/FBt1/detection_loss‚2.3796084875410255 0loss/FBt3/detection_loss‚0.48345520511567075 %loss/total_loss‚44.24476838272515 2loss/SRqqt3/assignment_loss‚0.8761286104737378 .loss/FRt4/detection_loss‚2.507868893444538 trainer/global_step‚5149 % _timestamp‚1.7884089044139426e+09 _runtime‚4007.401584515 0loss/FRt2/assignment_loss‚2.6986582474066663 .loss/FRt1/detection_loss‚2.890649251639843 1loss/SRqqt4/detection_loss‚1.7114956698590709 epoch‚4 /loss/FRt1/assignment_loss‚4.165632784366608 0loss/FRt3/assignment_loss‚1.5872365881999333 2loss/SRqqt1/assignment_loss‚2.0891983951583053 /loss/FRt3/detection_loss‚1.6824761490958433 .loss/FBt2/detection_loss‚2.397191325823466 1loss/FBt3/assignment_loss‚0.12878315943523144 1loss/SRqqt2/detection_loss‚1.5736782481627805 .loss/FBt4/detection_loss‚3.541697311401367;U^<Ù‚× 2loss/SRqqt2/assignment_loss‚1.7469079666770995 /loss/FBt4/assignment_loss‚0.790176491314196 _step‚423 1loss/SRqqt4/assignment_loss‚1.435134909986969 0loss/FBt2/assignment_loss‚0.1999394478««Vb„409613 1loss/SRqqt3/detection_loss‚1.0326587919201424 0loss/FRt4/assignment_loss‚3.3901491923765703 .loss/FRt2/detection_loss‚2.141174684350307 /loss/FBt1/detection_loss‚2.3796084875410255 0loss/FBt3/detection_loss‚0.48345520511567075 %loss/total_loss‚44.24476838272515 2loss/SRqqt3/assignment_loss‚0.8761286104737378 .loss/FRt4/detection_loss‚2.507868893444538 trainer/global_step‚5149 % _timestamp‚1.7884089044139426e+09 _runtime‚4007.401584515 0loss/FRt2/assignment_loss‚2.6986582474066663 .loss/FRt1/detection_loss‚2.890649251639843 1loss/SRqqt4/detection_loss‚1.7114956698590709 epoch‚4 /loss/FRt1/assignment_loss‚4.165632784366608 0loss/FRt3/assignment_loss‚1.5872365881999333 2loss/SRqqt1/assignment_loss‚2.0891983951583053 /loss/FRt3/detection_loss‚1.6824761490958433 .loss/FBt2/detection_loss‚2.397191325823466 1loss/FBt3/assignment_loss‚0.12878315943523144 1loss/SRqqt2/detection_loss‚1.5736782481627805 .loss/FBt4/detection_loss‚3.541697311401367 0loss/FBt1/assignment_loss‚0.4926990928339695 0loss/SRqqt1/detection_loss‚2.302169478300846§ßRa’ƒ‚(:‡ ÕèãÔ¶´¦ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.876' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·å žd„‚(:Ù ÕèãÔó©‡  disk.rbd1.out‚ 19.7802734375% disk./.usageGB‚247.83103561401367 disk.dm-1.out‚ 204.62109375 disk.sda.out‚ 204.62109375 disk.dm-0.in‚ 29.28515625 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 953.2265625( disk.nvme0c0n1.out‚35013.58447265625 disk.drbd1716.in‚0 network.sent‚ 1.1833733e+07 network.recv‚ 1.500129e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0& disk.nvme0n1.out‚35013.58447265625) proc.memory.percent‚17.10193157196045 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 204.62109375 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 29.28515625 cpu‚25.34917461923396 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 29.28515625$ memory_percent‚94.18196678161621# proc.memory.rssMB‚ 2801.98046875 disk.drbd1716.out‚0 disk.rbd1.in‚0* disk./.usagePercent‚16.632148744145823 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.sda1.in‚0# disk.dm-0.out‚35013.58447265625 disk.sda2.in‚0”IƒWœ…‚(:‘ äèãÔ•óš gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.41' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.681999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ3_†‚(:Ô äèãÔì¶Â  network.recv‚ 1.506668e+06 disk.sda.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 29.28515625 disk.dm-1.in‚0 disk.sda.out‚ 205.140625 disk.sda3.out‚ 205.140625 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 29.28515625& disk.nvme0n1.out‚35014.38916015625( proc.memory.availableMB‚ 953.21484375 disk.rbd0.out‚0* disk./.usagePercent‚16.632171016801305 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚35014.38916015625 disk.rbd1.in‚0 disk.dm-1.out‚ 205.140625) proc.memory.percent‚17.10193157196045% disk./.usageGB‚247.83136749267578 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd2080.out‚0 disk.rbd1.out‚ 19.7802734375# disk.dm-0.out‚35014.38916015625$ memory_percent‚94.18203830718994 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2801.98046875 disk.rbd0.in‚0 network.sent‚ 1.1941778e+07 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.drbd2265.in‚0 cpu‚25.34800165720522 proc.cpu.threads‚17WãâÞ2‡- _wandbruntime‚4042  _runtime‚4042ª6|®ÇˆÁ & lr-AdamW/pg2‚0.0001934569394901834 trainer/global_step‚5199 % _timestamp‚1.7884089394886985e+09 _runtime‚4042.476184637 _step‚424 & lr-AdamW/pg1‚0.00019345693949018348$KÌ‰Æ trainer/global_step‚5199 % _timestamp‚1.7884089394886985e+09 _runtime‚4042.476184637 _step‚424 & lr-AdamW/pg1‚0.0001934569394901834 & lr-AdamW/pg2‚0.0001934569394901834¨Õ® 2Š- _wandbruntime‚4042  _runtime‚40424À”§Ü'‹ÖO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24892212 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4567747957760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚5199 _step‚425 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302333881753088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330149176007168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306688216520192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173785622 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98151674 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298750182375616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4355009132544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75613910 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334717091783168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294411189691392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354352770048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98171712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306688216520192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69010520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 $ _timestamp‚1.788408939489465e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4355009132544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306688890885632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354352770048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44118310 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294411206730752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚4042.477141552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338975644864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75613910 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302333863750144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4567915776000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98151674 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24892651 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302333863750144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334716900024832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298749510898500ð‡G/á'ŒÛO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4355009132544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306688890885632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354352770048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44118310 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294411206730752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚4042.477141552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338975644864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75613910 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302333863750144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4567915776000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98151674 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24892651 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302333863750144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334716900024832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298749510898500 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24892212 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4567747957760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚5199 _step‚425 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302333881753088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330149176007168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306688216520192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173785622 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98151674 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298750182375616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4355009132544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75613910 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334717091783168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294411189691392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354352770048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98171712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306688216520192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69010520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 $ _timestamp‚1.788408939489465e+09©þݤ¨2- _wandbruntime‚4043  _runtime‚40438yâ‡JŽàM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚426 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75627926 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294466110861312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4339792080036 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173798052 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44126476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298805231464024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4355172197888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4355172197888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98190192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44126478 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69023570 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788408940164724e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75627924 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24897092 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302390216788992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173798052 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330210802804736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69023125 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚4043.152548667 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4355828573696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339137642072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334779408796160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98190192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75627924 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334779598444032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298805902941348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306745388986880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24896649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302390234791936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302390234791936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98170128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872¬\f¹ë&åM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306745388986880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24896649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302390234791936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302390234791936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98170128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚426 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75627926 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294466110861312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4339792080036 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173798052 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44126476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298805231464024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4355172197888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4355172197888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98190192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44126478 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69023570 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788408940164724e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75627924 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24897092 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302390216788992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173798052 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330210802804736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69023125 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚4043.152548667 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4355828573696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339137642072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334779408796160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98190192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75627924 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334779598444032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298805902941348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844ªõWN2- _wandbruntime‚4043  _runtime‚4043Þ{v1[‘Õ 1loss/FBt2/assignment_loss‚0.43426879472099245 &loss/total_loss‚42.062058486125736 0loss/FRt3/assignment_loss‚1.4865847871853755 0loss/SRqqt2/detection_loss‚1.977550357580185 0loss/SRqqt3/detection_loss‚0.992210170175408 /loss/FRt2/detection_loss‚2.3035987798048527 _runtime‚4043.153668067 1loss/FBt1/assignment_loss‚0.16170009227547175 -loss/FBt2/detection_loss‚2.59402746707201 epoch‚4 0loss/FRt1/assignment_loss‚3.5604903533540924 0loss/FRt2/assignment_loss‚3.6430402252138876 1loss/FBt3/assignment_loss‚0.20371441544774171 /loss/FRt1/detection_loss‚2.8314712479196746 % _timestamp‚1.7884089401663072e+09 1loss/SRqqt2/assignment_loss‚1.862000902378282 1loss/SRqqt4/detection_loss‚1.6369742775449947 .loss/FBt1/detection_loss‚2.616109960609012 /loss/FBt3/detection_loss‚0.5101612655321757 _step‚427 0loss/FBt4/assignment_loss‚0.6732909328836415 /loss/FRt4/detection_loss‚2.1218717405024696 /loss/FBt4/detection_loss‚2.0445259724344527 trainer/global_step‚5199 2loss/SRqqt1/assignment_loss‚1.5730944062982286 2loss/SRqqt3/assignment_loss‚0.8376741081108929 1loss/SRqqt4/assignment_loss‚1.521810539827055 /loss/FRt3/detection_loss‚1.6535200798310912 0loss/SRqqt1/detection_loss‚2.525091663002968 0loss/FRt4/assignment_loss‚2.2972759464207817ñé`’Ú 0loss/FRt4/assignment_loss‚2.2972759464207817 1loss/FBt2/assignment_loss‚0.43426879472099245 &loss/total_loss‚42.062058486125736 0loss/FRt3/assignment_loss‚1.4865847871853755 0loss/SRqqt2/detection_loss‚1.977550357580185 0loss/SRqqt3/detection_loss‚0.992210170175408 /loss/FRt2/detection_loss‚2.3035987798048527 _runtime‚4043.153668067 1loss/FBt1/assignment_loss‚0.16170009227547175 -loss/FBt2/detection_loss‚2.59402746707201 epoch‚4 0loss/FRt1/assignment_loss‚3.5604903533540924 0loss/FRt2/assignment_loss‚3.6430402252138876 1loss/FBt3/assignment_loss‚0.20371441544774171 /loss/FRt1/detection_loss‚2.8314712479196746 % _timestamp‚1.7884089401663072e+09 1loss/SRqqt2/assignment_loss‚1.862000902378282 1loss/SRqqt4/detection_loss‚1.6369742775449947 .loss/FBt1/detection_loss‚2.616109960609012 /loss/FBt3/detection_loss‚0.5101612655321757 _step‚427 0loss/FBt4/assignment_loss‚0.6732909328836415 /loss/FRt4/detection_loss‚2.1218717405024696 /loss/FBt4/detection_loss‚2.0445259724344527 trainer/global_step‚5199 2loss/SRqqt1/assignment_loss‚1.5730944062982286 2loss/SRqqt3/assignment_loss‚0.8376741081108929 1loss/SRqqt4/assignment_loss‚1.521810539827055 /loss/FRt3/detection_loss‚1.6535200798310912 0loss/SRqqt1/detection_loss‚2.525091663002968«ÜeaT’“‚(:‡ óèãÔÕŽ¿ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.484' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BúdÌS”‚(:È óèãÔÏï˜ % disk.nvme0n1.out‚35016.0712890625 disk.dm-1.out‚ 205.71875 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 29.28515625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda3.out‚ 205.71875 disk.drbd2080.in‚0" disk.dm-0.out‚35016.0712890625% disk./.usageGB‚247.83166885375977! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd1716.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.346921825671338 disk.dm-0.in‚ 29.28515625 disk.sda.out‚ 205.71875 disk.rbd1.out‚19.8125 network.sent‚ 1.1947611e+07' disk.nvme0c0n1.out‚35016.0712890625 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0$ memory_percent‚94.18370723724365 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2801.98046875) proc.memory.percent‚17.10193157196045 network.recv‚ 1.510777e+06) disk./.usagePercent‚16.63219124139651 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 952.94140625ŸÄ¦Üœ•‚(:‘ ‚éãÔ¨éñ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚153.15' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.630000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’]!õY–‚(:Î ‚éãÔ†¶é  disk.rbd1.out‚19.90625% disk.nvme0n1.out‚35016.8447265625 disk.dm-0.in‚ 29.28515625" disk.dm-0.out‚35016.8447265625 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 cpu‚25.34582498067862* disk./.usagePercent‚16.632236298722535 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2801.98046875 network.sent‚ 1.2056132e+07 network.recv‚ 1.517905e+06 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚35016.8447265625 disk.sda.out‚ 206.22265625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 29.28515625 disk.dm-1.out‚ 206.22265625$ memory_percent‚94.18220520019531 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0% disk./.usageGB‚247.83234024047852 disk.sda3.out‚ 206.22265625 disk.sda2.out‚0 disk.drbd1716.in‚0$ proc.memory.availableMB‚953.1875 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 29.28515625) proc.memory.percent‚17.10193157196045*§®¡2—- _wandbruntime‚4078  _runtime‚4078]ìUɘà ' lr-AdamW/pg1‚0.00019163735601474742 ' lr-AdamW/pg2‚0.00019163735601474742 trainer/global_step‚5249 % _timestamp‚1.7884089758254783e+09 _runtime‚4078.812922577 _step‚428æŠ& Î™È _step‚428 ' lr-AdamW/pg1‚0.00019163735601474742 ' lr-AdamW/pg2‚0.00019163735601474742 trainer/global_step‚5249 % _timestamp‚1.7884089758254783e+09 _runtime‚4078.812922577¬¨Sâß2š- _wandbruntime‚4078  _runtime‚4078±B´·Ü'›ÖO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76314708 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4395994938368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4379156546516 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚309546854268416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297156413258752 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175392434 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337842763274752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297156396219392 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99077726 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69647628 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788408975826174e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309546854268416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175412474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4078.813842261 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76314710 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99097764 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25120920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333231688950272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301536224243024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301535552765908 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305151533695488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611050384384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4395994938368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44526710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0÷8ÉDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788408975826174e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309546854268416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175412474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4078.813842261 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76314710 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99097764 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25120920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333231688950272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301536224243024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301535552765908 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_s"Qôotart/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305151533695488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611050384384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4395994938368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44526710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76314708 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4395994938368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4379156546516 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚309546854268416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297156413258752 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175392434 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337842763274752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297156396219392 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99077726 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69647628­ƒ™è×2- _wandbruntime‚4079  _runtime‚4079…Øä•ç&žáM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333293339687936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚297211300349952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚297211317389312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69660751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305207886734336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305207868731392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301591273331432 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76328724 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301591944808756 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309604026735104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333293363628032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚5249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4396814379520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚4079.501073931 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380627419444 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4396158003712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4396814379520 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76328726 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4396158003712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309604701113856 % _timestamp‚1.7884089765132484e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69660296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337905278641664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚297211317389312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69660751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305207886734336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305207868731392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301591273331432 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76328724 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301591944808756 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309604026735104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333293363628032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚5249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4396814379520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚4079.501073931 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380627419444 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4396158003712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4396814379520 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76328726 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4396158003712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309604701113856 % _timestamp‚1.7884089765132484e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69660296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337905278641664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336314225833472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4653906642944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177019286 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77015508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307969185637888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25349117 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304321594633316 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299901602747392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336314249773568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312405492016640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307969185637888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44935108 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25349551 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312406166382080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307969167634944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77015508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100003778 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299901619786752 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419991885924 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100003778 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307969167634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚4115.000694303 _step‚433 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340968132476416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77015510 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340968315846144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312406166382080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788409012013065e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70284661 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872×kÏáå'®ßO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336314225833472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4653906642944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177019286 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77015508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307969185637888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25349117 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304321594633316 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299901602747392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336314249773568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312405492016640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307969185637888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44935108 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25349551 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312406166382080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307969167634944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77015508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100003778 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299901619786752 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419991885924 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100003778 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307969167634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚4115.000694303 _step‚433 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340968132476416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77015510 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340968315846144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312406166382080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788409012013065e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70284661±U~• 2¯- _wandbruntime‚4115  _runtime‚4115mæ7Lê&°äM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341030628406784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304377315198840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25353656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308025520673792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4437800185344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437143809536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884090127022977e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100042296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437800185344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77029524 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100022232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308025520673792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312463338862080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70296932 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70297384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308025538676736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100022232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4654751835648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336375876571136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308025538676736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654921737728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299956506877952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177071822ÁùYc±éM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4437143809536 trainer/global_step‚5299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44943278 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177051756 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batÌZƒlŒ$ch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336375876571136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308025538676736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654921737728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299956506877952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177071822 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341030628406784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304377315198840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25353656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308025520673792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4437800185344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437143809536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884090127022977e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100042296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437800185344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77029524 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100022232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308025520673792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312463338862080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70296932 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70297384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308025538676736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100022232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4654751835648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P2¹- _wandbruntime‚4151  _runtime‚4151(©›ÉºÃ _step‚436 ' lr-AdamW/pg1‚0.00018796474263953273 ' lr-AdamW/pg2‚0.00018796474263953273 trainer/global_step‚5349 % _timestamp‚1.7884090485673397e+09 _runtime‚4151.5549026543N¸<Î»È ' lr-AdamW/pg2‚0.00018796474263953273 trainer/global_step‚5349 % _timestamp‚1.7884090485673397e+09 _runtime‚4151.554902654 _step‚436 ' lr-AdamW/pg1‚0.00018796474263953273´£;¦á2¼- _wandbruntime‚4151  _runtime‚4151×¼Ûá'½ÛO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315264129764864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307108307977840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884090485680923e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315264804130304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25577680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315264804130304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178666178 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45343508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100949868 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315264129764864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4477966550016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45343510 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310786819577344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344094010504192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302646809275392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310786819577344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339396786656768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4477310187520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77716310 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302646826314752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4460827225332 _runtime‚4151.555777734 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310786837580288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚425 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4477310187520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339396762716672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70921188 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚427 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4477966550016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178646138 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77716308 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77716310 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4697068612096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4461481663088 trainer/global_step‚5349¶qOXo¾àO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4477310187520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77716310 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302646826314752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4460827225332 _runtime‚4151.555777734 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310786837580288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚425 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4477310187520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339396762716672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70921188 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚427 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4477966550016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178646138 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77716308 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77716310 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4697068612096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4461481663088 trainer/global_step‚5349 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315264129764864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307108307977840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884090485680923e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315264804130304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25577680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315264804130304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178666178 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45343508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚ö_‡w 430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100949868 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315264129764864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4477966550016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45343510 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310786819577344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344094010504192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302646809275392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310786819577344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339396786656768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4478785991168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100968348 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70933846 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339458413454336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77730324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77730326 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45351678 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302701713405952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70934275 _runtime‚4152.242293091 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4461643660296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315321976610304 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582170 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339458437394432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302701730445312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4462298098260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307164028543572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178678608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4478785991168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310843190619136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4478129615360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344156523955200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310843172616192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315321976610304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344156332210176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚5349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 $ _timestamp‚1.788409049254491e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4698086560768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77730324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4478129615360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45351676 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307163357066248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178678608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25582597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310843190619136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚429 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178698674 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100948284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100968348 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315321302231552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310843172616192‡•´mï&ÁéM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100948284 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70933846 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339458413454336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77730324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77730326 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45351678 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302701713405952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70934275 _runtime‚4152.242293091 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4461643660296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315321976610304 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582170 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339458437394432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302701730445312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4462298098260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307164028543572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178678608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4478785991168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310843190619136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4478129615360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344156523955200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310843172616192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315321976610304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344156332210176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚5349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 $ _timestamp‚1.788409049254491e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4698086560768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77730324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4478129615360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45351676 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307163357066248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178678608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25582597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310843190619136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚429 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178698674 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100948284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100968348 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315321302231552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310843172616192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4478785991168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100968348 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318122767513088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347219404714496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101875920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71558445 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309893678368132 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚4187.865961694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318123441878528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313604489522688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25806960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4740105114624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚5399 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78417110 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313604489522688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313604471519744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45751910 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4740262447104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71558870 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4502317002496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313604471519744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4518952355840 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180272990 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 % _timestamp‚1.7884090848782022e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45751908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305392015803392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78417108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4518952355840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78417108 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305392032842752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚318123441878528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347219585987072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚441 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342479299599872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101875920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4518295993344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318122767513088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347219404714496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101875920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71558445 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309893678368132 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚4187.865961694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318123441878528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313604489522688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25806960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4740105114624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚5399 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78417110 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313604489522688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313604471519744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45751910 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4740262447104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71558870 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4502317002496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313604471519744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4518952355840 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180272990 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 % _timestamp‚1.7884090848782022e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45751908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305392015803392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78417108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4518952355840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78417108 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305392032842752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚318123441878528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347219585987072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚441 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342479299599872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928¹㯗æ2Ñ- _wandbruntime‚4188  _runtime‚4188ÂIœë&ÒåM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4519115421184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318180614358528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342540950337536 _step‚442 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4741128696320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4503133437668 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78431124 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347281917520384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884090857830665e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318179939979776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101894400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342540974277632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180325526 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180305460 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313660824558592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚5399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318180614358528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313660842561536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4502478999704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45760078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4519771796992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305446936973312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78431124 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519771796992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519115421184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71571619 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25811541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78431126 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101874336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71571192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45760076 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347282102973952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305446919933952 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4503133437668 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78431124 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347281917520384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884090857830665e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318179939979776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101894400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342540974277632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180325526 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180305460 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313660824558592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚5399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318180614358528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313660842561536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peñÐU¥ak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4502478999704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45760078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4519771796992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305446936973312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78431124 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519771796992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519115421184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71571619 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25811541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78431126 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101874336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71571192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45760076 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347282102973952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305446919933952 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320981405261312 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345561860423168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312679720235540 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4559281799168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46160310 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4559938161664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320981405261312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4559281799168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4542497904148 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320982079626752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4559938161664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4783533419520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102781934 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79117908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350345393842688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚4224.363793393 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72196268 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚445 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345561836483072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181899842 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5449 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320982079626752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26035506 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 % _timestamp‚1.7884091213759825e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72195814 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308137222331392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79117910 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79117908 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26035958 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4783363504128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320981405261312 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345561860423168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312679720235540 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4559281799168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46160310 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4559938161664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320981405261312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4559281799168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4542497904148 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320982079626752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4559938161664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4783533419520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102781934 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79117908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350345393842688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚4224.363793393 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72196268 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚445 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345561836483072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181899842 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5449 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320982079626752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26035506 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 % _timestamp‚1.7884091213759825e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72195814 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308137222331392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79117910 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79117908 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26035958 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4783363504128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72208609 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316478494503936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181932312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26040133 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350407892956672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4560757602816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72209073 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102800388 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚4225.051309779 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4560101227008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26040595 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345623487220736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4543314339112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4543968777076 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312736112278388 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321038577728000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79131926 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316478476500992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308192126461952 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4560757602816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181932312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312735440801064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102820452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321039252106752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345623511160832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884091220632458e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4560101227008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102820452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚446 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79131924 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102800388 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79131924zóHïð&ãêM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4543968777076 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312736112278388 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321038577728000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79131926 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316478476500992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308192126461952 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4560757602816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181932312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312735440801064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102820452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321039252106752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345623511160832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884091220632458e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4560101227008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102820452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚446 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79131924 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102800388 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79131924 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46168478 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350407699114496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316478494503936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚5449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79131926 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308192143501312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321039252106752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46168476 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72208609 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316478494503936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181932312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26040133 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350407892956672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4560757602816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72209073 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102800388 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚4225.051309779 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4560101227008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26040595 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345623487220736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4543314339112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430¾1ÝŠƒ2ä- _wandbruntime‚4225  _runtime‚4225KÓQ%WåÑ 2loss/SRqqt3/assignment_loss‚1.1203669351008203 -loss/FRt1/detection_loss‚2.87620997939791 /loss/FBt2/detection_loss‚1.4989325816814716 _step‚447 1loss/FBt3/assignment_loss‚0.11635447631957291 /loss/FRt1/assignment_loss‚4.022010135650635 /loss/FRt2/assignment_loss‚2.976955867281147 0loss/FBt2/assignment_loss‚0.6939171358513144 .loss/FRt3/detection_loss‚1.618293020612838 &loss/total_loss‚44.066565073404256 /loss/FRt4/assignment_loss‚2.650096978132541 2loss/SRqqt1/assignment_loss‚1.8455976432278043 1loss/SRqqt4/assignment_loss‚1.674345676837997 /loss/FRt4/detection_loss‚2.1940902603360324 .loss/FBt3/detection_loss‚0.512157427404187 /loss/FRt2/detection_loss‚2.1027446792406193 1loss/SRqqt2/detection_loss‚1.7258678061120651 _runtime‚4225.052588569 2loss/SRqqt2/assignment_loss‚1.3624394492921876 0loss/FBt4/assignment_loss‚0.3589384874794632 1loss/SRqqt4/detection_loss‚1.6486933011900295 .loss/FBt4/detection_loss‚4.669635698199272 % _timestamp‚1.7884091220652585e+09 0loss/FRt3/assignment_loss‚1.8353641702417742 1loss/FBt1/assignment_loss‚0.19022628009972525 1loss/SRqqt1/detection_loss‚3.0620834202993485 0loss/SRqqt3/detection_loss‚1.007479187699012 .loss/FBt1/detection_loss‚2.303764475716485 epoch‚4 trainer/global_step‚5449@¯kí\æÖ /loss/FRt4/assignment_loss‚2.650096978132541 2loss/SRqqt1/assignment_loss‚1.8455976432278043 1loss/SRqqt4/assignment_loss‚1.674345676837997 /loss/FRt4/detection_loss‚2.1940902603360324 .loss/FBt3/detection_loss‚0.512157427404187 /loss/FRt2/detection_loss‚2.1027446792406193 1loss/SRqqt2/detection_loss‚1.7258678061120651 _runtime‚4225.052588569 2loss/SRqqt2/assignment_loss‚1.3624394492921876 0loss/FBt4/assignment_loss‚0.3589384874794632 1loss/SRqqt4/detection_loss‚1.6486933011900295 .loss/FBt4/detection_loss‚4.669635698199272 % _timestamp‚1.7884091220652585e+09 0loss/FRt3/assignment_loss‚1.8353641702417742 1loss/FBt1/assignment_loss‚0.19022628009972525 1loss/SRqqt1/detection_loss‚3.0620834202993485 0loss/SRqqt3/detection_loss‚1.007479187699012 .loss/FBt1/detection_loss‚2.303764475716485 epoch‚4 trainer/global_step‚5449 2loss/SRqqt3/assignment_loss‚1.1203669351008203 -loss/FRt1/detection_loss‚2.87620997939791 /loss/FBt2/detection_loss‚1.4989325816814716 _step‚447 1loss/FBt3/assignment_loss‚0.11635447631957291 /loss/FRt1/assignment_loss‚4.022010135650635 /loss/FRt2/assignment_loss‚2.976955867281147 0loss/FBt2/assignment_loss‚0.6939171358513144 .loss/FRt3/detection_loss‚1.618293020612838 &loss/total_loss‚44.066565073404256¿Á4a~ç‚(:’ §êãÔ¤öŠ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.221' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.244200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AI>eeè‚(:Ú §êãÔ¿Äô  disk.sda2.in‚0 disk.dm-0.in‚ 29.28515625 disk.dm-1.out‚ 212.53515625# proc.memory.rssMB‚ 2801.98046875 cpu‚25.334188701810422 disk.sda3.in‚0& disk.nvme0n1.out‚35042.37353515625 network.recv‚ 1.575453e+06 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚16.633236776166346 disk.dm-1.in‚0 network.sent‚ 1.2531298e+07 disk.drbd1716.in‚0# disk.dm-0.out‚35042.37353515625 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 212.53515625 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 29.28515625( disk.nvme0c0n1.out‚35042.37353515625 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd0.in‚0$ memory_percent‚94.18303966522217! disk.nvme0c0n1.in‚ 29.28515625 disk.rbd1.out‚ 20.666015625 disk.sda3.out‚ 212.53515625 disk.drbd2266.in‚0 disk.drbd2266.out‚0) proc.memory.percent‚17.10193157196045 proc.cpu.threads‚17% disk./.usageGB‚247.84724807739258 disk.drbd2265.out‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 953.05078125ô¨Ò‘é‚(:† ¶êãÔÙ†ö gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.074' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4148 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ ÑœXê‚(:Í ¶êãÔÈÚÏ  disk.sda3.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 953.296875# proc.memory.rssMB‚ 2801.98046875 cpu‚25.333240056129753 disk.rbd1.out‚ 20.759765625 disk.sda.out‚ 212.96875 disk.drbd2080.in‚0 network.recv‚ 1.582837e+06 disk.dm-0.in‚ 29.28515625$ memory_percent‚94.18153762817383) disk./.usagePercent‚16.63329181778621 disk.sda2.in‚0( disk.nvme0c0n1.out‚35043.64892578125# disk.dm-0.out‚35043.64892578125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 29.28515625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 212.96875$ disk./.usageGB‚247.8480682373047 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 212.96875 disk.drbd2266.in‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd2265.in‚0 disk.drbd2266.out‚0) proc.memory.percent‚17.10193157196045 proc.cpu.threads‚17 network.sent‚ 1.2640053e+07& disk.nvme0n1.out‚35043.64892578125 disk.sda.in‚0(§’ë‚(:‡ ÅêãÔÙýü  gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.343' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.4686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÆnaì‚(:Ö ÅêãÔ”óµ  network.recv‚ 1.587111e+06 disk.sda.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 213.63671875 disk.sda1.in‚0) proc.memory.percent‚17.10193157196045 disk.dm-1.out‚ 213.63671875 disk.sda3.out‚ 213.63671875 disk.rbd1.out‚ 20.822265625 network.sent‚ 1.2646455e+07 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0* disk./.usagePercent‚16.633417005470452 disk.drbd2080.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚35047.0126953125 disk.nvme0n1.in‚ 29.28515625" disk.dm-0.out‚35047.0126953125# proc.memory.rssMB‚ 2801.98046875 disk.dm-1.in‚0 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 29.28515625% disk.nvme0n1.out‚35047.0126953125$ memory_percent‚94.18158531188965' proc.memory.availableMB‚ 953.2890625% disk./.usageGB‚247.84993362426758 disk.drbd2265.in‚0 disk.sda1.out‚0 cpu‚25.332167919121808 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 29.28515625 proc.cpu.threads‚17z@€2í- _wandbruntime‚4260  _runtime‚4260]¹90Éîà ' lr-AdamW/pg1‚0.00018237796331678564 ' lr-AdamW/pg2‚0.00018237796331678564 trainer/global_step‚5499 % _timestamp‚1.7884091577912357e+09 _runtime‚4260.778722277 _step‚448¼YÎïÈ _runtime‚4260.778722277 _step‚448 ' lr-AdamW/pg1‚0.00018237796331678564 ' lr-AdamW/pg2‚0.00018237796331678564 trainer/global_step‚5499 % _timestamp‚1.7884091577912357e+09ÀìR³f2ð- _wandbruntime‚4260  _runtime‚4260wo¾¦CñÚO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310882428859392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4600923967488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353470777337856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103707986 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚ 4260.77964216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323840043009536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚319239775404544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4600267604992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183546734 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚5499 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600923967488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72833363 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323840717374976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315466433580064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4826403971584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348644373366272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315465762102948 % _timestamp‚1.7884091577919908e+09 PCDej¡M>viceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79818710 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353470971193344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310882445898752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103707986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26265114 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4600267604992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79818708 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72833824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183526694 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26264655 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4826573886976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103728024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323840717374976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46568708 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319239793407488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319239793407488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79818708 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79818710 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323840043009536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚319239775404544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4600267604992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183546734 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚5499 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600923967488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72833363 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323840717374976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315466433580064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4826403971584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348644373366272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315465762102948 % _timestamp‚1.7884091577919908e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79818710 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353470971193344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310882445898752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103707986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26265114 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4600267604992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79818708 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72833824ÁH.J>2ó- _wandbruntime‚4261  _runtime‚4261 ™éëë&ôåM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4601087032832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103746504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348706048044032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827430836224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79832724 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103746504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚4261.466733716 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353533293426688 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310937350029312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚450 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183559164 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323897215476224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103726440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827269322752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4584149678520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79832726 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310937332989952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103726440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4601743408640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46576878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323897889854976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319296128443392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584804116484 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319296146446336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4601743408640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46576876 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315521482668472 % _timestamp‚1.7884091584786253e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319296146446336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79832724 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5499 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72846037 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79832726 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FõêM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310937350029312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚450 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183559164 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323897215476224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103726440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827269322752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4584149678520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79832726 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310937332989952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103726440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4601743408640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46576878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323897889854976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319296128443392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584804116484 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319296146446336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4601743408640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46576876 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315521482668472 % _timestamp‚1.7884091584786253e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319296146446336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79832724 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5499 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72846037 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceSíAøC² tatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79832726 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79832724 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103746504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚4261.466733716 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353533293426688 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414ÂÔ·áá2ö- _wandbruntime‚4261  _runtime‚4261…,æ[÷Õ 2loss/SRqqt2/assignment_loss‚1.3913491638947506 1loss/SRqqt1/detection_loss‚2.7478382322523327 1loss/SRqqt4/detection_loss‚1.9765479506357857 .loss/FBt1/detection_loss‚3.325652241706848 /loss/FBt3/detection_loss‚0.5861165041111808 /loss/FBt4/detection_loss‚2.5658334717154503 _runtime‚4261.468166056 _step‚451 0loss/FRt3/assignment_loss‚1.7046170566902785 0loss/FRt4/assignment_loss‚3.5457925862736173 1loss/FBt2/assignment_loss‚0.12736692605540156 1loss/FBt3/assignment_loss‚0.18618376745594278 .loss/FRt1/detection_loss‚3.061696654763715 /loss/FRt2/detection_loss‚2.2723307059361386 %loss/total_loss‚44.34227392932466 0loss/FRt2/assignment_loss‚2.8159217754235635 2loss/SRqqt4/assignment_loss‚1.1743823412967764 /loss/FRt3/detection_loss‚1.5723817672763691 /loss/FRt4/detection_loss‚2.6156365315119428 trainer/global_step‚5499 2loss/SRqqt3/assignment_loss‚0.9947260554887538 /loss/SRqqt2/detection_loss‚1.81646699990545 $ _timestamp‚1.788409158480812e+09 0loss/SRqqt3/detection_loss‚1.120429064712075 epoch‚4 0loss/FRt1/assignment_loss‚3.4894927740097046 0loss/FBt1/assignment_loss‚0.3216816180793103 1loss/SRqqt1/assignment_loss‚2.110694232362288 1loss/FBt4/assignment_loss‚0.15221240205028153 /loss/FBt2/detection_loss‚2.6669231057167053)“ƒk`øÚ 0loss/FBt1/assignment_loss‚0.3216816180793103 1loss/SRqqt1/assignment_loss‚2.110694232362288 1loss/FBt4/assignment_loss‚0.15221240205028153 /loss/FBt2/detection_loss‚2.6669231057167053 2loss/SRqqt2/assignment_loss‚1.3913491638947506 1loss/SRqqt1/detection_loss‚2.7478382322523327 1loss/SRqqt4/detection_loss‚1.9765479506357857 .loss/FBt1/detection_loss‚3.325652241706848 /loss/FBt3/detection_loss‚0.5861165041111808 /loss/FBt4/detection_loss‚2.5658334717154503 _runtime‚4261.468166056 _step‚451 0loss/FRt3/assignment_loss‚1.7046170566902785 0loss/FRt4/assignment_loss‚3.5457925862736173 1loss/FBt2/assignment_loss‚0.12736692605540156 1loss/FBt3/assignment_loss‚0.18618376745594278 .loss/FRt1/detection_loss‚3.061696654763715 /loss/FRt2/detection_loss‚2.2723307059361386 %loss/total_loss‚44.34227392932466 0loss/FRt2/assignment_loss‚2.8159217754235635 2loss/SRqqt4/assignment_loss‚1.1743823412967764 /loss/FRt3/detection_loss‚1.5723817672763691 /loss/FRt4/detection_loss‚2.6156365315119428 trainer/global_step‚5499 2loss/SRqqt3/assignment_loss‚0.9947260554887538 /loss/SRqqt2/detection_loss‚1.81646699990545 $ _timestamp‚1.788409158480812e+09 0loss/SRqqt3/detection_loss‚1.120429064712075 epoch‚4 0loss/FRt1/assignment_loss‚3.4894927740097046Ãe’"„’ù‚(:‡ ÔêãÔÙ¤£ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.607' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ÕÂ(^ú‚(:Ó ÔêãÔõùè  disk.sda.in‚0 disk.sda.out‚ 214.0546875 disk.drbd2266.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 953.2265625' disk.nvme0c0n1.out‚35047.9853515625 disk.rbd1.out‚ 20.822265625% disk.nvme0n1.out‚35047.9853515625 disk.dm-1.in‚0 disk.dm-1.out‚ 214.0546875 disk.drbd2265.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2801.98046875 disk.rbd0.in‚0 disk.rbd0.out‚0% disk./.usageGB‚247.85027694702148 disk.sda1.in‚0 network.sent‚ 1.2651738e+07 disk.drbd2080.in‚0 cpu‚25.331157414520142! disk.nvme0c0n1.in‚ 29.28515625 disk.sda2.in‚0$ memory_percent‚94.18196678161621) proc.memory.percent‚17.10193157196045 disk.nvme0n1.in‚ 29.28515625" disk.dm-0.out‚35047.9853515625 disk.sda3.out‚ 214.0546875 proc.cpu.threads‚17 disk.drbd2080.out‚0 network.recv‚ 1.591134e+06* disk./.usagePercent‚16.633440046148532 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 29.28515625 disk.drbd2265.out‚0j`*%’û‚(:‡ ãêãÔ®‚  gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚144.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.8344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^Ý`\ü‚(:Ñ ãêãÔôéô  disk.rbd1.out‚ 20.8544921875 disk.dm-1.out‚ 214.62109375 cpu‚25.330224716125354 disk.sda3.out‚ 214.62109375 disk.rbd0.in‚0 disk.drbd1716.in‚0' proc.memory.availableMB‚ 953.2265625 network.sent‚ 1.2760514e+07 disk.sda3.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚35050.13671875 disk.drbd2265.in‚0 disk.sda.out‚ 214.62109375# proc.memory.rssMB‚ 2801.98046875 proc.cpu.threads‚17 disk.sda1.out‚0% disk.nvme0c0n1.out‚35050.13671875* disk./.usagePercent‚16.633483311421816 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 29.28515625 disk.dm-0.out‚35050.13671875 disk.sda.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 29.28515625 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd2265.out‚0$ memory_percent‚94.18196678161621) proc.memory.percent‚17.10193157196045% disk./.usageGB‚247.85092163085938 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.recv‚ 1.598404e+06o 2¾2ý- _wandbruntime‚4296  _runtime‚4296h1€ÃÅþ¿ _step‚452 & lr-AdamW/pg1‚0.0001804964913266354 & lr-AdamW/pg2‚0.0001804964913266354 trainer/global_step‚5549 # _timestamp‚1.78840919390027e+09 _runtime‚4296.887733338%/»éÊÿÄ _runtime‚4296.887733338 _step‚452 & lr-AdamW/pg1‚0.0001804964913266354 & lr-AdamW/pg2‚0.0001804964913266354 trainer/global_step‚5549 # _timestamp‚1.78840919390027e+09ÄÀ €2€- _wandbruntime‚4296  _runtime‚4296)s÷à'ÚO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326699355123200 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326698680757760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185173586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80519508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322057427346944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4869653123584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚453 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104634038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322057445349888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104634038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46977110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4869493693952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4641909773312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4296.888706211 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104654076 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313627635387392 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80519508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26494442 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5549 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80519510 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318251803970356 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788409193901049e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313627652426752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356596587313152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641909773312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185153546 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351726934189568ò‡å'‚ßO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322057445349888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351726910249472 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73471552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356596587313152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641909773312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185153546 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351726934189568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326699355123200 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326698680757760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185173586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80519508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322057427346944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4869653123584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚453 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104634038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322057445349888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104634038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46977110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4869493693952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4641909773312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4296.888706211 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104654076 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313627635387392 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80519508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26494442 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5549 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80519510 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318251803970356 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788409193901049e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313627652426752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313682556557312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4642729214464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185206082 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326756527603200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4625639455892 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4624985017928 KEDeviceStatsMonitor.on_train_batch_eŠéÚßÂnd/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185186016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322113780385792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80533524 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚322113798388736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4870516912128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚454 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73484218 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4870347010048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322113798388736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73483736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318307524535880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46985278 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80533526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26498460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356659101839360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964L€ãð&…êM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚322113798388736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4870516912128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚454 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73484218 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4870347010048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322113798388736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73483736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318307524535880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46985278 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80533526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26498460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356659101839360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46985276 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26498940 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318308196013204 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313682556557312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4642729214464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185206082 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326756527603200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4625639455892 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4624985017928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185186016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322113780385792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80533524 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Æøé@2†- _wandbruntime‚4297  _runtime‚4297ì_ù[‡Õ 1loss/SRqqt3/detection_loss‚0.8884965200501849 /loss/FRt2/assignment_loss‚3.881509802341461 /loss/FRt4/assignment_loss‚2.981911160624944 1loss/FBt4/assignment_loss‚0.41975426192220766 /loss/FRt2/detection_loss‚2.2962779921293257 epoch‚4 trainer/global_step‚5549 2loss/SRqqt1/assignment_loss‚1.4314692549407482 % _timestamp‚1.7884091945964432e+09 .loss/FBt2/detection_loss‚2.720566166771783 _runtime‚4297.583824904 /loss/FRt1/assignment_loss‚3.672014129572901 1loss/FBt1/assignment_loss‚0.17032619454548695 1loss/SRqqt4/detection_loss‚1.9468653164043719 ,loss/FBt4/detection_loss‚2.6151282787323 _step‚455 2loss/SRqqt4/assignment_loss‚1.1822779671460086 1loss/FBt3/assignment_loss‚0.20123772113890082 .loss/FRt4/detection_loss‚2.592092682535832 1loss/SRqqt2/detection_loss‚2.3846941229908967 %loss/total_loss‚42.82523636628262 0loss/FRt3/assignment_loss‚1.2423578909799164 1loss/FBt2/assignment_loss‚0.12185046987401114 /loss/FRt3/detection_loss‚1.5342037803402133 1loss/SRqqt1/detection_loss‚2.5757251332203546 /loss/FRt1/detection_loss‚2.9757341393109025 /loss/FBt1/detection_loss‚2.5403572648763655 /loss/FBt3/detection_loss‚0.5927760966600892 2loss/SRqqt2/assignment_loss‚1.0041745085875655 2loss/SRqqt3/assignment_loss‚0.8534355105858453DŒ:Ð`ˆÚ 2loss/SRqqt4/assignment_loss‚1.1822779671460086 1loss/FBt3/assignment_loss‚0.20123772113890082 .loss/FRt4/detection_loss‚2.592092682535832 1loss/SRqqt2/detection_loss‚2.3846941229908967 %loss/total_loss‚42.82523636628262 0loss/FRt3/assignment_loss‚1.2423578909799164 1loss/FBt2/assignment_loss‚0.12185046987401114 /loss/FRt3/detection_loss‚1.5342037803402133 1loss/SRqqt1/detection_loss‚2.5757251332203546 /loss/FRt1/detection_loss‚2.9757341393109025 /loss/FBt1/detection_loss‚2.5403572648763655 /loss/FBt3/detection_loss‚0.5927760966600892 2loss/SRqqt2/assignment_loss‚1.0041745085875655 2loss/SRqqt3/assignment_loss‚0.8534355105858453 1loss/SRqqt3/detection_loss‚0.8884965200501849 /loss/FRt2/assignment_loss‚3.881509802341461 /loss/FRt4/assignment_loss‚2.981911160624944 1loss/FBt4/assignment_loss‚0.41975426192220766 /loss/FRt2/detection_loss‚2.2962779921293257 epoch‚4 trainer/global_step‚5549 2loss/SRqqt1/assignment_loss‚1.4314692549407482 % _timestamp‚1.7884091945964432e+09 .loss/FBt2/detection_loss‚2.720566166771783 _runtime‚4297.583824904 /loss/FRt1/assignment_loss‚3.672014129572901 1loss/FBt1/assignment_loss‚0.17032619454548695 1loss/SRqqt4/detection_loss‚1.9468653164043719 ,loss/FBt4/detection_loss‚2.6151282787323 _step‚455Ç ³K’‰‚(:‡ òêãÔì gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èŠ%WŠ‚(:Ì òêãÔÌêû  disk.nvme0n1.in‚ 29.28515625# disk.nvme0n1.out‚35051.04296875( proc.memory.availableMB‚ 953.00390625% disk./.usageGB‚247.85165786743164 cpu‚25.329207632296516 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.276625e+07* disk./.usagePercent‚16.633532720875927 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0) proc.memory.percent‚17.10195541381836 disk.sda3.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 215.1640625 disk.sda.in‚0$ memory_percent‚94.18332576751709 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2801.984375 network.recv‚ 1.603065e+06 disk.sda2.in‚0% disk.nvme0c0n1.out‚35051.04296875 disk.rbd1.in‚0 disk.dm-0.out‚35051.04296875 disk.dm-1.in‚0 disk.drbd2080.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.dm-0.in‚ 29.28515625 disk.sda.out‚ 215.1640625 disk.sda3.out‚ 215.1640625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 20.9482421875„Õnl’‹‚(:‡ ëãÔ¸‡ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.803' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1606 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßЇYŒ‚(:Î ëãÔÑ©»  disk.drbd2080.in‚0 disk.sda2.out‚0" disk.dm-0.out‚35054.2001953125 disk.nvme0n1.in‚ 29.28515625 disk.drbd2266.in‚0 disk.sda1.in‚0# memory_percent‚94.1852331161499 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0* disk./.usagePercent‚16.633567025885515% disk./.usageGB‚247.85216903686523 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 215.796875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 29.28515625 disk.dm-1.in‚0 disk.dm-1.out‚ 215.796875 disk.sda3.in‚0 disk.sda3.out‚ 215.796875) proc.memory.percent‚17.10195541381836 network.recv‚ 1.610365e+06! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd1716.out‚0 disk.sda.in‚0 disk.rbd0.out‚0 cpu‚25.32830743603607 disk.drbd2080.out‚0 network.sent‚ 1.2875276e+07 disk.rbd1.out‚ 21.0107421875% disk.nvme0n1.out‚35054.2001953125' disk.nvme0c0n1.out‚35054.2001953125( proc.memory.availableMB‚ 952.69140625! proc.memory.rssMB‚ 2801.984375¤2ˆ2- _wandbruntime‚4333  _runtime‚4333­¥äÄŽ¾ $ _timestamp‚1.788409230498722e+09 _runtime‚4333.486183877 _step‚456 % lr-AdamW/pg1‚0.000178606065085084 % lr-AdamW/pg2‚0.000178606065085084 trainer/global_step‚5599&â’Éà % lr-AdamW/pg2‚0.000178606065085084 trainer/global_step‚5599 $ _timestamp‚1.788409230498722e+09 _runtime‚4333.486183877 _step‚456 % lr-AdamW/pg1‚0.000178606065085084È[~n2- _wandbruntime‚4333  _runtime‚4333¼>ñ&:#‘ÛO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329557318505984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329557992871424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81220308 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105580128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81220310 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354809471072768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚316372858954752 trainer/global_step‚5599 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321038517314880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359722307098112 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324875079289344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81220310 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4682239216640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4665003922372 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105560090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105580128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186800438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74108877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26723820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47385508 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321037845837764 MGDeviceStatsMonitor.on_train_batch_start/reservãÞϧed_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47385510 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26723369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912836025344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4665658360128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324875079289344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚457 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359722119534080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496!'¤Œæ'’àO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4665003922372 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105560090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105580128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186800438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74108877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26723820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47385508 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321037845837764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47385510 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26723369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912836025344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4665658360128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324875079289344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚457 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359722119534080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329557318505984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329557992871424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81220308 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105580128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81220310 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354809471072768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚316372858954752 trainer/global_step‚5599 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321038517314880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359722307098112 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324875079289344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81220310 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4682239216640É^Ç[2“- _wandbruntime‚4334  _runtime‚4334«óžë&”åM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47393678 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74121995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913703804416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4665820357336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4666474795300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105598608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324931450331136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47393676 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4913548582400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81234324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26727866 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186812868 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329615165351424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81234326 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321094237880612 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324931450331136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4683058644480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321093566403288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74121542 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316427763085312 _step‚458 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81234324 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4683715020288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324931432328192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26728317 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105578544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316427746045952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359784646452736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354871097870336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚4334.176939375 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359784825614848 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105578544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4683715020288 trainer/global_step‚5599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329615165351424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324931432328192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884092311887636e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81234326 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324931450331136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4683058644480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321093566403288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74121542 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316427763085312 _step‚458 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81234324 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4683715020288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324931432328192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26728317 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105578544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316427746045952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359784646452736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354871097870336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚4334.176939375 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359784825614848 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105578544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4683715020288 trainer/global_step‚5599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329615165351424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324931432328192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884092311887636e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81234326 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74121995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913703804416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4665820357336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4666474795300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105598608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324931450331136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376ʃÇl„2–- _wandbruntime‚4334  _runtime‚4334"/é —Ô /loss/FRt1/assignment_loss‚4.261016194309507 0loss/FBt4/assignment_loss‚0.8607563114284338 .loss/FRt4/detection_loss‚2.669867515563965 0loss/SRqqt3/detection_loss‚1.143312431871891 epoch‚4 1loss/SRqqt4/assignment_loss‚1.667594137983244 1loss/FBt2/assignment_loss‚0.31505385010192793 /loss/FRt4/assignment_loss‚2.782279766187435 2loss/SRqqt1/assignment_loss‚2.1496830310891655 /loss/FRt3/detection_loss‚1.5371292520744295 /loss/FBt4/detection_loss‚2.9435031584330966 trainer/global_step‚5599 1loss/FBt1/assignment_loss‚0.18185483042725537 /loss/FRt1/detection_loss‚2.8397520588976994 /loss/FBt1/detection_loss‚3.1945300698280334 /loss/FBt3/detection_loss‚0.6185928543509785 /loss/FRt3/assignment_loss‚2.105785288489782 1loss/FBt3/assignment_loss‚0.12299904985370727 $ _timestamp‚1.788409231190623e+09 1loss/SRqqt4/detection_loss‚1.8188342121048051 %loss/total_loss‚46.06639048766614 _step‚459 2loss/SRqqt2/assignment_loss‚1.7723104982078075 2loss/SRqqt3/assignment_loss‚1.0090493415482342 /loss/FRt2/detection_loss‚2.2317449279288035 .loss/FBt2/detection_loss‚2.203393724229601 _runtime‚4334.178003622 0loss/FRt2/assignment_loss‚3.4345857171302145 1loss/SRqqt1µ4YÒW/detection_loss‚2.2577280191814197 1loss/SRqqt2/detection_loss‚1.9450342464447021¿åˆK_˜Ù 1loss/SRqqt2/detection_loss‚1.9450342464447021 /loss/FRt1/assignment_loss‚4.261016194309507 0loss/FBt4/assignment_loss‚0.8607563114284338 .loss/FRt4/detection_loss‚2.669867515563965 0loss/SRqqt3/detection_loss‚1.143312431871891 epoch‚4 1loss/SRqqt4/assignment_loss‚1.667594137983244 1loss/FBt2/assignment_loss‚0.31505385010192793 /loss/FRt4/assignment_loss‚2.782279766187435 2loss/SRqqt1/assignment_loss‚2.1496830310891655 /loss/FRt3/detection_loss‚1.5371292520744295 /loss/FBt4/detection_loss‚2.9435031584330966 trainer/global_step‚5599 1loss/FBt1/assignment_loss‚0.18185483042725537 /loss/FRt1/detection_loss‚2.8397520588976994 /loss/FBt1/detection_loss‚3.1945300698280334 /loss/FBt3/detection_loss‚0.6185928543509785 /loss/FRt3/assignment_loss‚2.105785288489782 1loss/FBt3/assignment_loss‚0.12299904985370727 $ _timestamp‚1.788409231190623e+09 1loss/SRqqt4/detection_loss‚1.8188342121048051 %loss/total_loss‚46.06639048766614 _step‚459 2loss/SRqqt2/assignment_loss‚1.7723104982078075 2loss/SRqqt3/assignment_loss‚1.0090493415482342 /loss/FRt2/detection_loss‚2.2317449279288035 .loss/FBt2/detection_loss‚2.203393724229601 _runtime‚4334.178003622 0loss/FRt2/assignment_loss‚3.4345857171302145 1loss/SRqqt1/detection_loss‚2.2577280191814197Ë£Vnú’™‚(:‡ ëãÔÛûÀ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.659' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7318 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“åèPš‚(:Å ëãÔ×›˜  disk.dm-1.out‚216.3125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda.in‚0 network.sent‚ 1.2880396e+07 disk.sda2.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚35055.3603515625 network.recv‚ 1.61331e+06' disk.nvme0c0n1.out‚35055.3603515625 disk.drbd1716.in‚0 disk.rbd1.out‚ 21.0107421875 disk.nvme0n1.in‚ 29.28515625 disk.drbd2265.in‚0) proc.memory.percent‚17.10195541381836 disk.rbd0.out‚0! proc.memory.rssMB‚ 2801.984375 disk.sda.out‚216.3125 disk.sda3.out‚216.3125 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 29.28515625) disk./.usagePercent‚16.63359697876702$ disk./.usageGB‚247.8526153564453 disk.sda1.in‚0 disk.sda2.out‚0" disk.dm-0.out‚35055.3603515625# memory_percent‚94.1838026046753 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0( proc.memory.availableMB‚ 952.92578125 cpu‚25.32728378025279 disk.drbd1716.out‚0 disk.dm-0.in‚ 29.28515625ÄÑ¡Ì‘›‚(:† ŸëãÔÏã¹ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.517 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø÷-˜]œ‚(:Ò ŸëãÔ§…  disk.drbd2266.out‚0 network.sent‚ 1.2988297e+07 disk.drbd2265.in‚0 disk.sda1.out‚0 cpu‚25.326277051903745 disk.drbd1716.out‚0 disk.dm-0.in‚ 29.28515625 disk.sda3.out‚ 216.92578125( disk./.usagePercent‚16.6336282116862 disk.dm-1.out‚ 216.92578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 216.92578125 disk.drbd2080.out‚0( proc.memory.availableMB‚ 953.25390625 disk.drbd2080.in‚0' disk.nvme0c0n1.out‚35059.5517578125 disk.drbd1716.in‚0 disk.rbd1.out‚ 21.10546875 disk.nvme0n1.in‚ 29.28515625" disk.dm-0.out‚35059.5517578125 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2801.984375 network.recv‚ 1.620385e+06! disk.nvme0c0n1.in‚ 29.28515625 disk.sda.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.10195541381836 proc.cpu.threads‚17% disk.nvme0n1.out‚35059.5517578125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0% disk./.usageGB‚247.85308074951172$ memory_percent‚94.18179988861084±»Ž†œ‚(:‘ ®ëãÔ£øÀ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.782' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.756400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯®¬ûXž‚(:Í ®ëãÔ÷è * disk./.usagePercent‚16.633756983475923 disk.dm-1.out‚ 217.62109375 disk.drbd2265.out‚0 disk.sda.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0$ memory_percent‚94.18115615844727& proc.memory.availableMB‚ 953.359375 network.sent‚ 1.2994295e+07 network.recv‚ 1.624888e+06 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 21.19921875 cpu‚25.3253933925792 proc.cpu.threads‚17$ disk.nvme0n1.out‚35060.603515625 disk.dm-0.in‚ 29.28515625 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 29.28515625! disk.dm-0.out‚35060.603515625 disk.dm-1.in‚0 disk.sda3.out‚ 217.62109375 disk.rbd1.in‚0) proc.memory.percent‚17.10195541381836 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2801.984375% disk./.usageGB‚247.85499954223633 disk.drbd2265.in‚0 disk.sda.out‚ 217.62109375 disk.sda2.in‚0& disk.nvme0c0n1.out‚35060.603515625 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.sda3.in‚0þŒg’2Ÿ- _wandbruntime‚4369  _runtime‚4369t`ÁÇ Á % _timestamp‚1.7884092668253443e+09 _runtime‚4369.812873178 _step‚460 & lr-AdamW/pg1‚0.0001767070785541268 & lr-AdamW/pg2‚0.0001767070785541268 trainer/global_step‚5649ôü“ÍÌ¡Æ _step‚460 & lr-AdamW/pg1‚0.0001767070785541268 & lr-AdamW/pg2‚0.0001767070785541268 trainer/global_step‚5649 % _timestamp‚1.7884092668253443e+09 _runtime‚4369.812873178̺šž2¢- _wandbruntime‚4369  _runtime‚4369¸€má'£ÛO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332415956254208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106506180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4723881384960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357892007955968 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319118048443392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332416630619648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319118065482752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327692731231744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327692731231744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106506180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26952054 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332415956254208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332416630619648 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332415956254208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332416630619648 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4723881384960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357892007955968 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319118048443392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332416630619648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319118065482752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327692731231744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327692731231744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106506180ÍÎDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4724700826112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188439720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74758994 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327749084270592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106524660 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319172952573952 _step‚462 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47802078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47802076 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323879608270696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚332473803099648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4956843849216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74758540 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327749084270592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956669752832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4724700826112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106524660 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188459786 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106504596 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357953634753536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319172969613312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.largwgíhÅe_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4724044450304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357953658693632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362910304506368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323880279748020 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81935124 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4724044450304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362910502542848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4706655696744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚5649 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4370.505263286 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884092675173168e+09 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327749102273536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332473803099648 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81935124 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4724044450304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362910502542848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4706655696744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚5649 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4370.505263286 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884092675173168e+09 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327749102273536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332473803099648 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4724700826112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188439720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74758994 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327749084270592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106524660 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319172952573952 _step‚462 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47802078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47802076 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323879608270696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚332473803099648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4956843849216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74758540 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327749084270592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956669752832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4724700826112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106524660 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188459786 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106504596 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357953634753536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319172969613312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4724044450304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357953658693632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362910304506368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323880279748020 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752ÎLÈý2¨- _wandbruntime‚4370  _runtime‚4370@´D”Y©Ó /loss/SRqqt1/detection_loss‚2.74397617989573 1loss/SRqqt3/detection_loss‚0.7609549384858263 &loss/total_loss‚42.889887473588885 epoch‚4 trainer/global_step‚5649 /loss/FRt4/detection_loss‚2.6400072792681253 1loss/SRqqt4/detection_loss‚1.8127425276535623 _step‚463 /loss/FRt4/assignment_loss‚3.400464505684085 2loss/SRqqt2/assignment_loss‚1.4151592079211366 1loss/FBt1/assignment_loss‚0.29864196571923624 /loss/FRt3/detection_loss‚1.8480391038476296 /loss/FRt1/detection_loss‚2.9247730489700072 .loss/FRt2/detection_loss‚2.127739153154538 0loss/FRt2/assignment_loss‚2.9510622296644295 2loss/SRqqt3/assignment_loss‚1.0423267629536752 2loss/SRqqt4/assignment_loss‚1.6399028279045795 0loss/FBt2/assignment_loss‚0.3044092536435983 /loss/FRt3/assignment_loss‚1.467645545050783 0loss/FBt3/assignment_loss‚0.1789702380420252 .loss/FBt2/detection_loss‚2.894379962235689 /loss/FBt3/detection_loss‚0.4530239978184303 /loss/FBt4/detection_loss‚2.1683897897601128 _runtime‚4370.506841294 0loss/FBt4/assignment_loss‚0.3139269309467636 % _timestamp‚1.7884092675194604e+09 1loss/SRqqt2/detection_loss‚1.6769512181932276 .loss/FBt1/detection_loss‚2.368191962892359 0loss/FRt1/assignment_loss‚3.3921163928124214 1loss/SRqqt1/assignment_loss‚2.066092451070917­*¼Ã^ªØ /loss/FRt3/assignment_loss‚1.467645545050783 0loss/FBt3/assignment_loss‚0.1789702380420252 .loss/FBt2/detection_loss‚2.894379962235689 /loss/FBt3/detection_loss‚0.4530239978184303 /loss/FBt4/detection_loss‚2.1683897897601128 _runtime‚4370.506841294 0loss/FBt4/assignment_loss‚0.3139269309467636 % _timestamp‚1.7884092675194604e+09 1loss/SRqqt2/detection_loss‚1.6769512181932276 .loss/FBt1/detection_loss‚2.368191962892359 0loss/FRt1/assignment_loss‚3.3921163928124214 1loss/SRqqt1/assignment_loss‚2.066092451070917 /loss/SRqqt1/detection_loss‚2.74397617989573 1loss/SRqqt3/detection_loss‚0.7609549384858263 &loss/total_loss‚42.889887473588885 epoch‚4 trainer/global_step‚5649 /loss/FRt4/detection_loss‚2.6400072792681253 1loss/SRqqt4/detection_loss‚1.8127425276535623 _step‚463 /loss/FRt4/assignment_loss‚3.400464505684085 2loss/SRqqt2/assignment_loss‚1.4151592079211366 1loss/FBt1/assignment_loss‚0.29864196571923624 /loss/FRt3/detection_loss‚1.8480391038476296 /loss/FRt1/detection_loss‚2.9247730489700072 .loss/FRt2/detection_loss‚2.127739153154538 0loss/FRt2/assignment_loss‚2.9510622296644295 2loss/SRqqt3/assignment_loss‚1.0423267629536752 2loss/SRqqt4/assignment_loss‚1.6399028279045795 0loss/FBt2/assignment_loss‚0.3044092536435983ÏWèÀ«‚(:’ ½ëãÔ³¦¢ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.111599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇA—9\¬‚(:Ñ ½ëãÔ¢ú²  disk.sda3.in‚0 disk.sda2.in‚0$ memory_percent‚94.18137073516846( proc.memory.availableMB‚ 953.32421875 cpu‚25.3244367950806 network.recv‚ 1.6294e+06 disk.dm-0.in‚ 29.28515625# disk.dm-0.out‚35064.39404296875 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 218.23828125 disk.drbd2266.in‚0 disk.rbd1.out‚ 21.26171875! proc.memory.rssMB‚ 2801.984375$ disk./.usageGB‚247.8555030822754 disk.sda.out‚ 218.23828125 disk.sda3.out‚ 218.23828125 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.10195541381836 disk.nvme0n1.in‚ 29.28515625 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 29.28515625( disk.nvme0c0n1.out‚35064.39404296875 disk.drbd1716.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚16.63379077647044 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 network.sent‚ 1.3000504e+07& disk.nvme0n1.out‚35064.39404296875Ðì¼Z’­‚(:‡ ÌëãÔŒ§ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.343' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.4686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\]œ-x®‚(:í ÌëãÔ„“ú  disk.nvme0n1.in‚ 29.28515625! proc.memory.rssMB‚ 2801.984375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 218.71484375 disk.sda.out‚ 218.71484375) proc.memory.percent‚17.10195541381836 network.sent‚ 1.3108528e+07) disk./.usagePercent‚16.63382508148003 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd1716.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚35065.28466796875 disk.dm-0.in‚ 29.28515625# disk.dm-0.out‚35065.28466796875 disk.drbd2265.out‚0( disk.nvme0c0n1.out‚35065.28466796875 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125$ memory_percent‚94.18139457702637 cpu‚25.323454195415437 proc.cpu.threads‚17 disk.sda3.out‚ 218.71484375 disk.drbd2266.out‚0 disk.drbd2080.in‚0' proc.memory.availableMB‚ 953.3203125 disk.sda1.in‚0 disk.rbd1.out‚ 21.26171875 disk.drbd2265.in‚0 network.recv‚ 1.636057e+06% disk./.usageGB‚247.85601425170898 disk.sda3.in‚ 0.1953125 disk.sda2.in‚0 Ì2¯- _wandbruntime‚4406  _runtime‚4406ÀÅmÇ°Á trainer/global_step‚5699 % _timestamp‚1.7884093032816563e+09 _runtime‚4406.269125445 _step‚464 & lr-AdamW/pg1‚0.0001747999274797106 & lr-AdamW/pg2‚0.0001747999274797106÷#‹2Ì±Æ & lr-AdamW/pg1‚0.0001747999274797106 & lr-AdamW/pg2‚0.0001747999274797106 trainer/global_step‚5699 % _timestamp‚1.7884093032816563e+09 _runtime‚4406.269125445 _step‚464Ðò!ô*2²- _wandbruntime‚4406  _runtime‚4406ëCäÄá'³ÛO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107432232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321863272010752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4746674601188 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82621910 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82621908 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330510383174144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82621910 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335275268367872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75384397 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335274594002432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335274594002432 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _step‚465 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107412194 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326609929572580 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321863254971392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190034102 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4764867190784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326610601049696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4999186647552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚330510383174144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4764867190784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48202310 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330510401177088 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚360974544839168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27182089 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107432232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48202308 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330510401177088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4747329038944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82621908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚365973541825536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚365973731486720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _runtime‚4406.270042594 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999020926464 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335275268367872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4764210828288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75384860 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884093032823706e+09Üʪ´àO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330510401177088 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚360974544839168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27182089 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107432232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48202308 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330510401177088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4747329038944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82621908 VHDevX oÈ iceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚365973541825536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚365973731486720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _runtime‚4406.270042594 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999020926464 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335275268367872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4764210828288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75384860 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884093032823706e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107432232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321863272010752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4746674601188 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82621910 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82621908 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330510383174144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82621910 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335275268367872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75384397 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335274594002432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335274594002432 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _step‚465 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107412194 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326609929572580 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321863254971392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190034102 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4764867190784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326610601049696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4999186647552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚330510383174144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4764867190784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48202310 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5699Ñ àž02µ- _wandbruntime‚4406  _runtime‚4406ÊF ¹ë&¶åM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321918176141312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190066572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190066572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335331766469120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330566736212992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5000044095488 _runtime‚4406.949492574 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366036041635840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107450712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚466 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361036195576832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5699 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4747491036152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27186702 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361036171636736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82635924 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4765030256128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4765030256128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326666321615428 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 % _timestamp‚1.7884093039616768e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330566736212992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107430648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚485 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190086638 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107430648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75397178 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27187187 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321918159101952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335332440847872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚330566754215936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82635924 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4748145474116 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190066572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190066572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335331766469120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330566736212992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5000044095488 _runtime‚4406.949492574 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366036041635840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107450712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚466 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361036195576832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5699 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4747491036152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27186702 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361036171636736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82635924 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4765030256128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4765030256128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326666321615428 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 % _timestamp‚1.7884093039616768e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330566736212992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107430648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚485 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190086638 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107430648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75397178 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27187187 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321918159101952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335332440847872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Ò‹ÒN2¸- _wandbruntime‚4406  _runtime‚4406uÇx\¹Ö 1loss/FBt3/assignment_loss‚0.06915454164224277 trainer/global_step‚5699 _runtime‚4406.951139209 .loss/FRt2/detection_loss‚2.161866828804928 .loss/FBt1/detection_loss‚2.498121048013369 epoch‚4 1loss/SRqqt3/detection_loss‚0.9641793901505677 _step‚467 /loss/FRt1/assignment_loss‚3.549204781137664 /loss/FRt2/assignment_loss‚3.253648487634437 0loss/FRt3/assignment_loss‚1.4935390639763613 1loss/FBt1/assignment_loss‚0.13453965770167997 .loss/FRt4/detection_loss‚2.460137441754341 1loss/SRqqt2/detection_loss‚1.9577607892605724 .loss/FBt3/detection_loss‚0.731578258459922 .loss/FBt4/detection_loss‚2.549214392900467 2loss/SRqqt2/assignment_loss‚1.5001127878013922 1loss/FBt4/assignment_loss‚0.14679775293916464 /loss/FBt2/detection_loss‚3.2845487713813784 &loss/total_loss‚43.689147031151336 % _timestamp‚1.7884093039637597e+09 0loss/FRt4/assignment_loss‚2.6935993999242784 2loss/SRqqt3/assignment_loss‚1.0986515960075718 .loss/FRt1/detection_loss‚3.131173775113862 1loss/SRqqt1/detection_loss‚2.5597034795530913 1loss/SRqqt4/detection_loss‚1.8002570140780063 2loss/SRqqt1/assignment_loss‚1.5573164574031173 2loss/SRqqt4/assignment_loss‚1.8383267458136145 1loss/FBt2/assignment_loss‚0.38309593200683595 /loss/FRt3/detection_loss‚1.8726186376924698N ÂtaºÛ 1loss/SRqqt2/detection_loss‚1.9577607892605724 .loss/FBt3/detection_loss‚0.731578258459922 .loss/FBt4/detection_loss‚2.549214392900467 2loss/SRqqt2/assignment_loss‚1.5001127878013922 1loss/FBt4/assignment_loss‚0.14679775293916464 /loss/FBt2/detection_loss‚3.2845487713813784 &loss/total_loss‚43.689147031151336 % _timestamp‚1.7884093039637597e+09 0loss/FRt4/assignment_loss‚2.6935993999242784 2loss/SRqqt3/assignment_loss‚1.0986515960075718 .loss/FRt1/detection_loss‚3.131173775113862 1loss/SRqqt1/detection_loss‚2.5597034795530913 1loss/SRqqt4/detection_loss‚1.8002570140780063 2loss/SRqqt1/assignment_loss‚1.5573164574031173 2loss/SRqqt4/assignment_loss‚1.8383267458136145 1loss/FBt2/assignment_loss‚0.38309593200683595 /loss/FRt3/detection_loss‚1.8726186376924698 1loss/FBt3/assignment_loss‚0.06915454164224277 trainer/global_step‚5699 _runtime‚4406.951139209 .loss/FRt2/detection_loss‚2.161866828804928 .loss/FBt1/detection_loss‚2.498121048013369 epoch‚4 1loss/SRqqt3/detection_loss‚0.9641793901505677 _step‚467 /loss/FRt1/assignment_loss‚3.549204781137664 /loss/FRt2/assignment_loss‚3.253648487634437 0loss/FRt3/assignment_loss‚1.4935390639763613 1loss/FBt1/assignment_loss‚0.13453965770167997 .loss/FRt4/detection_loss‚2.460137441754341ÓV&H‘»‚(:† ÛëãÔ„øó gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.315' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.863 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý›O5l¼‚(:ñ ÛëãÔŒ‚  network.sent‚ 1.3114405e+07 disk.rbd1.in‚0 disk.sda3.out‚ 219.41015625 network.recv‚ 1.640208e+06 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 29.28515625 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 219.41015625 disk.sda.out‚ 219.41015625 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 21.2939453125( proc.memory.availableMB‚ 953.32421875 cpu‚25.322552351128788 proc.cpu.threads‚17% disk./.usageGB‚247.85682678222656 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚94.18137073516846! proc.memory.rssMB‚ 2801.984375( disk.nvme0c0n1.out‚35067.09326171875 disk.dm-0.in‚ 29.28515625 disk.sda2.in‚0* disk./.usagePercent‚16.633879611084822# disk.dm-0.out‚35067.09326171875 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚35067.09326171875 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0) proc.memory.percent‚1‘hÕ‘7.10195541381836.îòÃ’½‚(:‡ êëãԶ̚ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.659' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7318 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò1ïo¾‚(:ä êëãÔú´þ % disk./.usageGB‚247.85748291015625 disk.rbd1.out‚ 21.4736328125 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125# disk.dm-0.out‚35068.05029296875 disk.rbd0.out‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 29.28515625! proc.memory.rssMB‚ 2801.984375 disk.dm-1.out‚ 219.984375& disk.nvme0n1.out‚35068.05029296875 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 219.984375 disk.drbd2080.in‚0 disk.sda2.out‚0 cpu‚25.321622191253702 disk.dm-0.in‚ 29.28515625 disk.sda.out‚ 219.984375 disk.sda2.in‚0$ memory_percent‚94.18182373046875 network.sent‚ 1.3222961e+07) disk./.usagePercent‚16.63392364438071 disk.drbd1716.out‚0 disk.drbd2080.out‚0" proc.memory.availableMB‚953.25) proc.memory.percent‚17.10195541381836 network.recv‚ 1.646965e+06 disk.nvme0n1.in‚ 29.28515625 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚35068.05029296875 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.drbd2266.in‚0ãÛ-œ’¿‚(:‡ ùëãÔ±õ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.876' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôûtýjÀ‚(:ß ùëãÔŽ‚Ý % disk./.usageGB‚247.85809326171875 disk.sda.in‚ 0.1953125 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 29.28515625! disk.dm-0.out‚35070.845703125" memory_percent‚94.183349609375! proc.memory.rssMB‚ 2801.984375 disk.rbd1.in‚0 network.sent‚ 1.3229311e+07 disk.dm-1.out‚ 220.64453125 disk.rbd0.out‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 29.28515625 network.recv‚ 1.651694e+06 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda.out‚ 220.64453125 disk.sda3.out‚ 220.64453125 disk.drbd2080.in‚0 disk.rbd1.out‚ 21.5361328125) proc.memory.percent‚17.10195541381836 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚35070.845703125) disk./.usagePercent‚16.63396460558619 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.320712672138466 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚35070.845703125 disk.dm-0.in‚ 29.28515625 proc.memory.availableMB‚953 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0ˆ¡¥"2Á- _wandbruntime‚4442  _runtime‚4442XDßÔÈ $ _timestamp‚1.788409339375453e+09 _runtime‚4442.362878405 _step‚468 ' lr-AdamW/pg1‚0.00017288500930926076 ' lr-AdamW/pg2‚0.00017288500930926076 trainer/global_step‚5749s£¿ÍÃÇ ' lr-AdamW/pg1‚0.00017288500930926076 ' lr-AdamW/pg2‚0.00017288500930926076 trainer/global_step‚5749 $ _timestamp‚1.788409339375453e+09 _runtime‚4442.362878405 _step‚468Ô8ý2Ä- _wandbruntime‚4442  _runtime‚4442>ü25à'ÅÚO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4805852996608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27412650 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.788409339376152e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333328035116544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83322710 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76023360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5042273750528 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206438400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108358284 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48610710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182498304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329395971439988 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4788164378352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191660954 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338133906116096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4805852996608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4787509940596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333328053119488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83322710 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108338246 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364057081722368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108338246 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191680994 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚4442.363837704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329396642917104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333328053119488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338133906116096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338133231750656 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369099149034496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚469 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83322708 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333328035116544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4805196634112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83322708 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76022885 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338133231750656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108358284 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27412177 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369099355472896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364057057782272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5042091252224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324608478538752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872öCîþå'ÆßO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108358284 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27412177 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369099355472896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364057057782272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5042091252224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324608478538752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4805852996608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27412650 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.788409339376152e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333328035116544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83322710 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76023360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5042273750528 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206438400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108358284 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48610710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182498304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329395971439988 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4788164378352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191660954 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338133906116096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4805852996608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4787509940596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333328053119488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83322710 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108338246 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364057081722368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108338246 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191680994 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚4442.363837704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329396642917104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333328053119488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338133906116096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338133231750656 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369099149034496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚469 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83322708 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333328035116544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4805196634112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83322708 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76022885 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338133231750656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Õ°D÷¥2Ç- _wandbruntime‚4443  _runtime‚4443yK<•¨!ÈåM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4806672437760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4788980813524 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364118732460032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108376764 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚4443.040718143 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83336724 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108356700 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108356700 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76035570 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5043113683968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83336724 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329451692005512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚470 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4806672437760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4806016061952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333384406158336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338190404217344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48618878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27416694 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83336726 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4806016061952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369161650204672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191693424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5042941684736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369161846144000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4788326375560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191713490 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364118708519936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108376764 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitØ ÿhCor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338191078596096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329452363482836 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360=Ç Êð&ÉêM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83336724 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108356700 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108356700 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76035570 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5043113683968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83336724 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329451692005512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚470 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4806672437760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4806016061952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333384406158336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338190404217344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48618878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27416694 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83336726 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4806016061952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369161650204672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191693424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5042941684736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369161846144000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4788326375560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191713490 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364118708519936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108376764 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338191078596096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329452363482836 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4806672437760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4788980813524 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364118732460032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108376764 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚4443.040718143Ö2°»Û2Ê- _wandbruntime‚4443  _runtime‚4443ú‘ ‹^ËØ %loss/total_loss‚43.13337130902043 2loss/SRqqt1/assignment_loss‚1.8755933069463433 1loss/FBt1/assignment_loss‚0.32195049476078047 1loss/FBt2/assignment_loss‚0.08404245274222275 /loss/FRt2/detection_loss‚2.2470411267545503 1loss/SRqqt4/detection_loss‚1.8241139464080334 .loss/FBt3/detection_loss‚0.560230225402241 trainer/global_step‚5749 0loss/FRt1/assignment_loss‚3.2111770120160332 0loss/FRt2/assignment_loss‚2.5868670715226068 2loss/SRqqt3/assignment_loss‚0.8738919038101504 1loss/SRqqt3/detection_loss‚1.3097729062977828 .loss/FBt1/detection_loss‚2.890641144343785 /loss/FBt3/assignment_loss‚0.192320129762051 _runtime‚4443.041830871 2loss/SRqqt4/assignment_loss‚1.6348315527263497 0loss/FBt4/assignment_loss‚0.0749212818827516 /loss/FRt4/detection_loss‚2.3605933785438538 1loss/SRqqt2/detection_loss‚1.8650833150992792 epoch‚4 % _timestamp‚1.7884093400544565e+09 2loss/SRqqt2/assignment_loss‚1.5030665094188105 /loss/FBt2/detection_loss‚2.7832538419299655 _step‚471 0loss/FRt3/assignment_loss‚1.3145651618639629 0loss/FRt4/assignment_loss‚3.5396482759051855 /loss/FRt1/detection_loss‚3.0270730688654144 /loss/FRt3/detection_loss‚1.8209562786751323 1loss/SRqqt1/detection_loss‚2.6080431218804985 /loss/FBt4/detection_loss‚2.6236938014626503ˆabcÌÝ 2loss/SRqqt4/assignment_loss‚1.6348315527263497 0loss/FBt4/assignment_loss‚0.0749212818827516 /loss/FRt4/detection_loss‚2.3605933785438538 1loss/SRqqt2/detection_loss‚1.8650833150992792 epoch‚4 % _timestamp‚1.7884093400544565e+09 2loss/SRqqt2/assignment_loss‚1.5030665094188105 /loss/FBt2/detection_loss‚2.7832538419299655 _step‚471 0loss/FRt3/assignment_loss‚1.3145651618639629 0loss/FRt4/assignment_loss‚3.5396482759051855 /loss/FRt1/detection_loss‚3.0270730688654144 /loss/FRt3/detection_loss‚1.8209562786751323 1loss/SRqqt1/detection_loss‚2.6080431218804985 /loss/FBt4/detection_loss‚2.6236938014626503 %loss/total_loss‚43.13337130902043 2loss/SRqqt1/assignment_loss‚1.8755933069463433 1loss/FBt1/assignment_loss‚0.32195049476078047 1loss/FBt2/assignment_loss‚0.08404245274222275 /loss/FRt2/detection_loss‚2.2470411267545503 1loss/SRqqt4/detection_loss‚1.8241139464080334 .loss/FBt3/detection_loss‚0.560230225402241 trainer/global_step‚5749 0loss/FRt1/assignment_loss‚3.2111770120160332 0loss/FRt2/assignment_loss‚2.5868670715226068 2loss/SRqqt3/assignment_loss‚0.8738919038101504 1loss/SRqqt3/detection_loss‚1.3097729062977828 .loss/FBt1/detection_loss‚2.890641144343785 /loss/FBt3/assignment_loss‚0.192320129762051 _runtime‚4443.041830871׊Œ˜¤Í‚(:’ ˆìãÔ¬ú gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.561' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.912200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qˆý®q΂(:æ ˆìãÔÍÿú  disk.sda3.in‚ 0.1953125 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 29.28515625 disk.nvme0n1.in‚ 29.28515625 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2801.984375& disk.nvme0c0n1.out‚35071.794921875 disk.rbd1.in‚0 disk.rbd1.out‚ 21.6220703125& proc.memory.availableMB‚ 952.703125 cpu‚25.31980368294341 network.recv‚ 1.655724e+06 disk.dm-0.in‚ 29.28515625! disk.dm-0.out‚35071.794921875 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 221.08984375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.sent‚ 1.3234685e+07 disk.sda1.out‚0$ disk.nvme0n1.out‚35071.794921875 disk.drbd2265.in‚0$ disk./.usageGB‚247.8587303161621 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1953125$ memory_percent‚94.18516159057617 disk.sda.out‚ 221.08984375) disk./.usagePercent‚16.63400735884441 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 221.08984375) proc.memory.percent‚17.10195541381836 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0$®Ñ<Ï‚(:’ —ìãÔ™šÙ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.145' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.229000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ßÑÿxЂ(:í —ìãÔ”©‹  disk.dm-0.in‚ 29.28515625 disk.dm-1.out‚ 221.77734375 disk.drbd2265.out‚0 disk.sda3.out‚ 221.77734375 disk.dm-1.in‚ 0.1953125( proc.memory.availableMB‚ 952.57421875 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 29.28515625( disk.nvme0c0n1.out‚35075.17236328125 disk.nvme0n1.in‚ 29.28515625 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0# disk.dm-0.out‚35075.17236328125 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0! proc.memory.rssMB‚ 2801.984375$ disk./.usageGB‚247.8590850830078 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 221.77734375 disk.drbd2266.out‚0 cpu‚25.31889002110534 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2266.in‚0# memory_percent‚94.1859483718872* disk./.usagePercent‚16.634031167545093 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 21.6845703125& disk.nvme0n1.out‚35075.17236328125) proc.memory.percent‚17.10195541381836 network.sent‚ 1.334334e+07 network.recv‚ 1.662678e+06 disk.rbd0.in‚0 disk.drbd1716.out‚0õåé]2Ñ- _wandbruntime‚4478  _runtime‚4478:_f·ÆÒÀ _runtime‚4478.880918433 _step‚472 & lr-AdamW/pg1‚0.0001709627231088539 & lr-AdamW/pg2‚0.0001709627231088539 trainer/global_step‚5799 $ _timestamp‚1.788409375893476e+09¶bm;ËÓÅ trainer/global_step‚5799 $ _timestamp‚1.788409375893476e+09 _runtime‚4478.880918433 _step‚472 & lr-AdamW/pg1‚0.0001709627231088539 & lr-AdamW/pg2‚0.0001709627231088539Øi|B‚2Ô- _wandbruntime‚4478  _runtime‚4478VÔÆðà'ÕÚO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336145687058944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109264298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27641422 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327353668027392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84023510 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846182439936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372225050231296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340992543864320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367139618605568 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚473 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4478.881968614 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84023508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4846182439936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5085278487552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340991869498880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109264298 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4846838802432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49019110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367139594665472 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340991869498880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372224873153024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193307846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27641875 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327353685066752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76660985 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4828345280004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193287806 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76660530 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4828999717760 $ _timestamp‚1.788409375894311e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332182684784512 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332182013307396 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84023508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49019108 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336145687058944ôƒýyÖßO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4478.881968614 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84023508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4846182439936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5085278487552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340991869498880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109264298 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4846838802432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49019110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367139594665472 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340991869498880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372224873153024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193307846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27641875 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327353685066752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76660985 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4828345280004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193287806 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76660530 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4828Ë=c‹l999717760 $ _timestamp‚1.788409375894311e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332182684784512 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332182013307396 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84023508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49019108 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336145687058944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336145687058944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109264298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27641422 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327353668027392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84023510 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846182439936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372225050231296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340992543864320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367139618605568 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚473ÙÍ»Ú2×- _wandbruntime‚4479  _runtime‚4479ª5xë&ØåM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76673278 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372287372402176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332238405350244 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341049716344320 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76673739 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327408572157952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49027278 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193320276 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27646461 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109302816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5086126999040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84037524 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336202040097792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109282752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336202058100736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327408589197312 trainer/global_step‚5799 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847001867776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430l#ð&ÙêM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27646002 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341049716344320 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336202040097792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49027276 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193340342 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193320276 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27646461 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109302816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5086126999040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84037524 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336202040097792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109282752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336202058100736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327408589197312 trainer/global_step‚5799 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847001867776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76673278 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372287372402176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332238405350244 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341049716344320 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76673739 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327408572157952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49027278 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49427508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887824608256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370222155488768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77298258 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4887824608256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84724310 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375350746508288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788409412063495e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338963339001344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343851181612544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194934698 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110190350 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5128433687040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4887168245760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343850507247104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚477 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84724310 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4887168245760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110210388 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110210388 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343850507247104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5128591019520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚338963357004288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756oˆr«å'èßO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194914658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚5849 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27870305 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110210388 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343850507247104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5128591019520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚338963357004288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚4515.051188215 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77297813 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375350565235712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370222131548672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4869180619412 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338963357004288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334968055174804 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110190350 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49427510 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49427508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887824608256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370222155488768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77298258 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4887824608256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84724310 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375350746508288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788409412063495e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338963339001344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343851181612544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194934698 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110190350 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5128433687040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4887168245760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343850507247104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚477 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84724310 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4887168245760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110210388 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335023775740328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888644049408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888644049408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370283806226432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚478 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887987673600 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5129283354624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330153795725312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194947128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4870651492340 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343908354092544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339019692040192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4869997054376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84738326 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27875334 trainer/global_step‚5849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343907679713792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375413257385984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110228868 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5129451159552 % _timestamp‚1.7884094127437792e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339019692040192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4887987673600 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110208804 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49435678 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77310558 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330153778685952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194967194 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343908354092544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888644049408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888644049408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370283806226432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚478 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887987673600 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5129283354624 ^JDeviceStatsMonitor.on_train_Æorù batch_end/requested_bytes.large_pool.allocated‚330153795725312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194947128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4870651492340 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343908354092544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339019692040192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4869997054376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84738326 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27875334 trainer/global_step‚5849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343907679713792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375413257385984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110228868 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5129451159552 % _timestamp‚1.7884094127437792e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339019692040192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4887987673600 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110208804 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49435678 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77310558 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330153778685952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194967194 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343908354092544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335023775740328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346709144995328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346709819360768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332844081083392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111116402 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337754097042212 trainer/global_step‚5899 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49835908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378476393800704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4928810414080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928810414080 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378476583461888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4910015958820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77935322 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111136440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5171891089920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5171725368832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85425108 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28099869 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196541510 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4928154051584 _runtime‚4551.481624074 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85425110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346709819360768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111136440 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85425110 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373304692371968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚481 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196561550 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚481 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196561550 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346709144995328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346709819360768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332844081083392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111116402 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337754097042212 trainer/global_step‚5899 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inact_e«# ive_split.large_pool.freed‚49835908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378476393800704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4928810414080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928810414080 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378476583461888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4910015958820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77935322 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111136440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5171891089920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5171725368832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85425108 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28099869 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196541510 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4928154051584 _runtime‚4551.481624074áxØa2ù- _wandbruntime‚4552  _runtime‚4552F ì¹é&úãM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77947977 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196573980 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5899 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196573980 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111134856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49844076 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373366319169536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5172584129536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111154920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341837343982592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346766991840768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346766991840768 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378539095044096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332898985213952 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85439124 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4929629855232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111154920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196594046 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚4552.166691759 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111134856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378538903299072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373366343109632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337810489085060 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4929629855232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 # _timestamp‚1.78840944917885e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346766317462016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5172751934464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85439124 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928973479424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77948452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332899002253312 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85439126 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341837361985536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4910832393784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414ñÒâ¸î&ûèM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111134856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378538903299072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373366343109632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337810489085060 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4929629855232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 # _timestamp‚1.78840944917885e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346766317462016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5172751934464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85439124 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928973479424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77948452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332899002253312 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85439126 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341837361985536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4910832393784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77947977 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196573980 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5899 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196573980 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111134856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49844076 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373366319169536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5172584129536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111154920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341837343982592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346766991840768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346766991840768 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378539095044096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332898985213952 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85439124 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4929629855232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111154920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196594046 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚4552.166691759 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430âä‹s¾2ü- _wandbruntime‚4552  _runtime‚4552¡8•Ø_ýÙ 2loss/SRqqt2/assignment_loss‚1.8905411123953484 2loss/FBt4/assignment_loss‚0.006602915159115004 .loss/FBt4/detection_loss‚2.644025464852651 epoch‚4 % _timestamp‚1.7884094491803272e+09 .loss/FRt2/detection_loss‚2.321672265612802 0loss/FRt2/assignment_loss‚2.6814170978790104 1loss/FBt2/assignment_loss‚0.12646226535798633 1loss/FBt3/assignment_loss‚0.13726292708588161 /loss/FBt3/detection_loss‚0.4805623846204609 /loss/FRt1/detection_loss‚2.9121424042516284 /loss/FRt4/detection_loss‚2.3712266595526175 1loss/SRqqt2/detection_loss‚1.7617007709211774 _step‚483 /loss/FRt4/assignment_loss‚3.178089602427049 2loss/SRqqt4/assignment_loss‚1.7556303397945638 /loss/FRt3/detection_loss‚1.7006839881837368 %loss/total_loss‚44.47673250603433 trainer/global_step‚5899 1loss/SRqqt1/detection_loss‚2.4119141439596814 0loss/SRqqt3/detection_loss‚1.085457554415745 /loss/FBt2/detection_loss‚2.9970448836684227 _runtime‚4552.168017729 /loss/FRt3/assignment_loss‚2.055419887304306 2loss/SRqqt3/assignment_loss‚1.3107298083932084 /loss/FBt1/detection_loss‚2.9213384687900543 0loss/FRt1/assignment_loss‚3.5144709712929196 1loss/FBt1/assignment_loss‚0.30276275145297404 1loss/SRqqt4/detection_loss‚1.7468059046088524 2loss/SRqqt1/assignment_loss‚2.16276793405413642kædþÞ .loss/FRt2/detection_loss‚2.321672265612802 0loss/FRt2/assignment_loss‚2.6814170978790104 1loss/FBt2/assignment_loss‚0.12646226535798633 1loss/FBt3/assignment_loss‚0.13726292708588161 /loss/FBt3/detection_loss‚0.4805623846204609 /loss/FRt1/detection_loss‚2.9121424042516284 /loss/FRt4/detection_loss‚2.3712266595526175 1loss/SRqqt2/detection_loss‚1.7617007709211774 _step‚483 /loss/FRt4/assignment_loss‚3.178089602427049 2loss/SRqqt4/assignment_loss‚1.7556303397945638 /loss/FRt3/detection_loss‚1.7006839881837368 %loss/total_loss‚44.47673250603433 trainer/global_step‚5899 1loss/SRqqt1/detection_loss‚2.4119141439596814 0loss/SRqqt3/detection_loss‚1.085457554415745 /loss/FBt2/detection_loss‚2.9970448836684227 _runtime‚4552.168017729 /loss/FRt3/assignment_loss‚2.055419887304306 2loss/SRqqt3/assignment_loss‚1.3107298083932084 /loss/FBt1/detection_loss‚2.9213384687900543 0loss/FRt1/assignment_loss‚3.5144709712929196 1loss/FBt1/assignment_loss‚0.30276275145297404 1loss/SRqqt4/detection_loss‚1.7468059046088524 2loss/SRqqt1/assignment_loss‚2.1627679340541364 2loss/SRqqt2/assignment_loss‚1.8905411123953484 2loss/FBt4/assignment_loss‚0.006602915159115004 .loss/FBt4/detection_loss‚2.644025464852651 epoch‚4 % _timestamp‚1.7884094491803272e+09ãL+(uÿ‚(:„ ñìãÔ«•š gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.55' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.31 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04hŽ…y€‚(:î ñìãÔÊÔ“  network.sent‚ 1.3584819e+07# disk.dm-0.out‚35087.05810546875 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 225.72265625! proc.memory.rssMB‚ 2801.984375% disk./.usageGB‚247.86596298217773 disk.drbd2266.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚35087.05810546875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.sda1.in‚0) disk./.usagePercent‚16.63449274912933 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 225.72265625 disk.drbd2265.out‚0) proc.memory.percent‚17.10195541381836 disk.drbd2080.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.rbd1.out‚ 22.115234375 cpu‚25.313459007441754 network.recv‚ 1.693933e+06 disk.drbd2080.out‚0' proc.memory.availableMB‚ 952.6484375 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 29.28515625& disk.nvme0n1.out‚35087.05810546875 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.sda.out‚ 225.72265625 proc.cpu.threads‚17 disk.dm-0.in‚ 29.28515625$ memory_percent‚94.18549537658691ÒEݦ’‚(:‡ €íãÔÐîÀ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ”t‚‚(:é €íãÔº¢  disk.rbd0.in‚0 network.sent‚ 1.3692759e+07 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 226.15234375 disk.nvme0n1.in‚ 29.28515625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 29.28515625$ memory_percent‚94.18482780456543 disk.sda1.out‚0' disk.nvme0c0n1.out‚35088.1416015625" disk.dm-0.out‚35088.1416015625 cpu‚25.31265297036068 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 22.115234375 disk.drbd2266.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda.out‚ 226.15234375 proc.cpu.threads‚17 disk.sda1.in‚0' proc.memory.availableMB‚ 952.7578125% disk./.usageGB‚247.86660385131836 disk.drbd2265.in‚0 disk.sda3.out‚ 226.15234375! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd1716.out‚0) proc.memory.percent‚17.10195541381836 network.recv‚ 1.70057e+06) disk./.usagePercent‚16.63453575839508 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.rbd1.in‚0% disk.nvme0n1.out‚35088.1416015625! proc.memory.rssMB‚ 2801.984375ßnx2ƒ- _wandbruntime‚4587  _runtime‚4587:„ MÈ„Â trainer/global_step‚5949 $ _timestamp‚1.788409484499249e+09 _runtime‚4587.486707649 _step‚484 ' lr-AdamW/pg1‚0.00016515566951974146 ' lr-AdamW/pg2‚0.00016515566951974146Nû*òÍ…Ç ' lr-AdamW/pg1‚0.00016515566951974146 ' lr-AdamW/pg2‚0.00016515566951974146 trainer/global_step‚5949 $ _timestamp‚1.788409484499249e+09 _runtime‚4587.486707649 _step‚484ä†Fŧ2†- _wandbruntime‚4587  _runtime‚4587«,Ê5h‡ÚO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198168362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚381602184768000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86125908 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349568457108992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚381602003495424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376387229255168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4969796219904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340540138909620 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28329427 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112042454 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969796219904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112042454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5214798180352 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚344598642886144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚344598660889088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335589304650752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4951505735984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚485 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.cur†<ä¤xrent‚ 181272576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335589287611392 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50244310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4950851298228 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349567782743552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _runtime‚4587.487603448 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112062492 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86125908 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78573737 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198188402 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 $ _timestamp‚1.788409484499975e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969139857408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚5949 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349567782743552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0#‘W2å'ˆßO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349567782743552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198168362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚381602184768000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86125908 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349568457108992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚381602003495424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376387229255168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4969796219904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340540138909620 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28329427 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112042454 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969796219904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112042454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5214798180352 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚344598642886144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚344598660889088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335589304650752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4951505735984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚485 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335589287611392 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50244310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4950851298228 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349567782743552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _runtime‚4587.487603448 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112062492 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86125908 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78573737 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198188402 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 $ _timestamp‚1.788409484499975e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969139857408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚5949 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560åÓ‡hð2‰- _wandbruntime‚4588  _runtime‚4588:’ü‚ë&ŠåM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344655013927936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198200832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198200832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112060908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381664497229312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376448856052736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349624955210240 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198220898 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78585990 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4970615661056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5215815272960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112080972 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4970615661056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86139926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50252478 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340595859475144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚5949 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376448879992832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚486 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78586453 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112060908 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344655013927936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚4588.393486991 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4951667733192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28333514 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335644191741952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340596530952468 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4969959285248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884094854056697e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112080972 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4952322171156 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349625629588992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50252476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86139926 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349625629588992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86139924 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4969959285248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381664695265792 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28333975 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344654995924992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78585990 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4970615661056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5215815272960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112080972 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4970615661056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86139926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50252478 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340595859475144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚5949 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376448879992832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚486 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78586453 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112060908 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344655013927936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚4588.393486991 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4951667733192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28333514 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335644191741952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340596530952468 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4969959285248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884094854056697e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112080972 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4952322171156 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349625629588992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50252476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚861›‹m¶39926 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349625629588992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86139924 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4969959285248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381664695265792 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28333975 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344654995924992 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344655013927936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198200832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198200832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112060908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381664497229312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376448856052736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928æOÃ/2Œ- _wandbruntime‚4588  _runtime‚4588—ÃM¬\Ö 2loss/SRqqt4/assignment_loss‚1.4854417864978313 1loss/FBt2/assignment_loss‚0.03933512925868854 .loss/FRt1/detection_loss‚2.947045036156972 .loss/FBt2/detection_loss‚2.707286238670349 % _timestamp‚1.7884094854072418e+09 0loss/FRt2/assignment_loss‚3.5474277635415397 2loss/SRqqt3/assignment_loss‚0.8344409973023162 1loss/SRqqt4/detection_loss‚1.8427401608228684 %loss/total_loss‚40.89328344283227 2loss/SRqqt1/assignment_loss‚1.6753231766431227 1loss/FBt1/assignment_loss‚0.09447916890106474 1loss/FBt3/assignment_loss‚0.14855347676351635 .loss/FRt4/detection_loss‚2.244606058423718 /loss/FBt4/detection_loss‚2.9451104521751406 1loss/SRqqt2/detection_loss‚1.9289408071952707 .loss/FBt1/detection_loss‚2.134175881743431 /loss/FBt3/detection_loss‚0.3771431366830845 trainer/global_step‚5949 _runtime‚4588.394599699 /loss/FRt2/detection_loss‚1.9178101742923597 /loss/FRt1/assignment_loss‚3.397501069307327 epoch‚4 0loss/FRt4/assignment_loss‚2.1325948337713876 1loss/FBt4/assignment_loss‚0.09119530720636249 1loss/SRqqt2/assignment_loss‚1.483476837037825 /loss/FRt3/detection_loss‚1.4088112335238192 1loss/SRqqt3/detection_loss‚1.0668320747065372 _step‚487 0loss/FRt3/assignment_loss‚1.4732883550502636 0loss/SRqqt1/detection_loss‚2.9697242871574735ÕJaŽÛ 2loss/SRqqt4/assignment_loss‚1.4854417864978313 1loss/FBt2/assignment_loss‚0.03933512925868854 .loss/FRt1/detection_loss‚2.947045036156972 .loss/FBt2/detection_loss‚2.707286238670349 % _timestamp‚1.7884094854072418e+09 0loss/FRt2/assignment_loss‚3.5474277635415397 2loss/SRqqt3/assignment_loss‚0.8344409973023162 1loss/SRqqt4/detection_loss‚1.8427401608228684 %loss/total_loss‚40.89328344283227 2loss/SRqqt1/assignment_loss‚1.6753231766431227 1loss/FBt1/assignment_loss‚0.09447916890106474 1loss/FBt3/assignment_loss‚0.14855347676351635 .loss/FRt4/detection_loss‚2.244606058423718 /loss/FBt4/detection_loss‚2.9451104521751406 1loss/SRqqt2/detection_loss‚1.9289408071952707 .loss/FBt1/detection_loss‚2.134175881743431 /loss/FBt3/detection_loss‚0.3771431366830845 trainer/global_step‚5949 _runtime‚4588.394599699 /loss/FRt2/detection_loss‚1.9178101742923597 /loss/FRt1/assignment_loss‚3.397501069307327 epoch‚4 0loss/FRt4/assignment_loss‚2.1325948337713876 1loss/FBt4/assignment_loss‚0.09119530720636249 1loss/SRqqt2/assignment_loss‚1.483476837037825 /loss/FRt3/detection_loss‚1.4088112335238192 1loss/SRqqt3/detection_loss‚1.0668320747065372 _step‚487 0loss/FRt3/assignment_loss‚1.4732883550502636 0loss/SRqqt1/detection_loss‚2.969724287157473ç¬Ûõe‘‚(:† íãÔÒï™ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.517 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û¬‚ø{‚(:ð íãÔìžß  disk.rbd0.out‚0 disk.drbd1716.out‚0* disk./.usagePercent‚16.634575439562887 disk.sda1.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚35091.02880859375) proc.memory.percent‚17.10195541381836 cpu‚25.311751131413946 network.recv‚ 1.704744e+06 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚35091.02880859375 disk.drbd2265.out‚0 disk.drbd2266.in‚0$ memory_percent‚94.18845176696777 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.nvme0n1.in‚ 29.28515625# disk.dm-0.out‚35091.02880859375 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 226.69140625 network.sent‚ 1.3698828e+07% disk./.usageGB‚247.86719512939453 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 226.69140625 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 22.1474609375 disk.sda.in‚ 0.1953125 disk.sda.out‚ 226.69140625' proc.memory.availableMB‚ 952.1640625 disk.sda2.out‚0 disk.dm-0.in‚ 29.28515625! proc.memory.rssMB‚ 2801.984375 proc.cpu.threads‚17"Ѥ‘‚(:’ žíãÔ¯½– gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.052' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.610400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/\>¡z’‚(:ï žíãÔ̱„  network.sent‚ 1.3807309e+07 disk.sda3.in‚ 0.1953125 disk.rbd1.out‚ 22.3271484375 disk.nvme0n1.in‚ 29.28515625 disk.dm-1.out‚ 227.38671875# memory_percent‚94.1856861114502* disk./.usagePercent‚16.634622288941653 disk.drbd2080.out‚0# disk.dm-0.out‚35091.91943359375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚35091.91943359375) proc.memory.percent‚17.10195541381836 network.recv‚ 1.711968e+06 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.sda.out‚ 227.38671875 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚35091.91943359375 disk.drbd2265.out‚0' proc.memory.availableMB‚ 952.6171875 disk.sda3.out‚ 227.38671875 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚25.310888446699387 proc.cpu.threads‚17% disk./.usageGB‚247.86789321899414! disk.nvme0c0n1.in‚ 29.28515625 disk.dm-0.in‚ 29.28515625! proc.memory.rssMB‚ 2801.984375 disk.sda.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0ÜDÜ4’“‚(:‡ ­íãÔ­¾× gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Km{”‚(:ð ­íãÔµ°‰  disk.nvme0n1.in‚ 29.28515625 disk.drbd2265.out‚0' proc.memory.availableMB‚ 952.6015625) proc.memory.percent‚17.10195541381836* disk./.usagePercent‚16.634654801898503 disk.sda2.in‚0( disk.nvme0c0n1.out‚35093.73974609375 disk.rbd1.in‚0 network.sent‚ 1.3813615e+07% disk./.usageGB‚247.86837768554688 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 29.28515625 disk.sda3.out‚ 228.33984375 disk.drbd2265.in‚0 cpu‚25.310056855447183 disk.sda.out‚ 228.33984375 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 disk.rbd1.out‚ 22.3896484375$ memory_percent‚94.18578147888184 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2801.984375# disk.dm-0.out‚35093.73974609375 network.recv‚ 1.716432e+06! disk.nvme0c0n1.in‚ 29.28515625& disk.nvme0n1.out‚35093.73974609375 disk.dm-1.out‚ 228.33984375 proc.cpu.threads‚17W1„<2•- _wandbruntime‚4623  _runtime‚4623}í¶É–à ' lr-AdamW/pg1‚0.00016320793131593047 ' lr-AdamW/pg2‚0.00016320793131593047 trainer/global_step‚5999 % _timestamp‚1.7884095207099924e+09 _runtime‚4623.697485335 _step‚488FuZ6Î—È ' lr-AdamW/pg1‚0.00016320793131593047 ' lr-AdamW/pg2‚0.00016320793131593047 trainer/global_step‚5999 % _timestamp‚1.7884095207099924e+09 _runtime‚4623.697485335 _step‚488èÕÅÈB2˜- _wandbruntime‚4623  _runtime‚4623RXò^á'™ÛO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4623.698384841 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010125663232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352427094857216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384727518447104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199795214 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79210060 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343326852254144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86826710 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚352427094857216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4992341075392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347416294828544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86826710 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28557787 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5010125663232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338334511178752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5257935678464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86826708 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112968506 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352426420491776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352426420491776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379469742198272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5257776248832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚489 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112988544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379469766138368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86826708 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5010782025728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4991686637636 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚5999 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347416294828544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50652708 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347416312831488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384727701816832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338334494139392òJîæ'šàO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5257776248832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚489 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112988544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379469766138368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86826708 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5010782025728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4991686637636 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚5999 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347416294828544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50652708 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347416312831488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384727701816832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338334494139392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4623.698384841 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010125663232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352427094857216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384727518447104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199795214 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79210060 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343326852254144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86826710 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚352427094857216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4992341075392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347416294828544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86826710 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28557787 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5010125663232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338334511178752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5257935678464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86826708 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112968506 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352426420491776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352426420491776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379469742198272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048é‘qw2›- _wandbruntime‚4624  _runtime‚4624 #/âœäM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5011601466880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_spli­%HÌ&t.large_pool.peak‚30 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79222647 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379531392935936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4992503072600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79223075 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384790052710912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343381901342552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4993157510564 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5258806608384 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 170773504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚379531416876032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011601466880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347472647867392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352484267337216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352484267337216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50660878 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199827684 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50660876 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5258659774976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28562197 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347472665870336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343382572819876 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86840724 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010945091072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352483592958464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347472665870336 $ _timestamp‚1.788409521391835e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347472647867392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010945091072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384790223484416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79222647 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379531392935936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4992503072600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79223075 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384790052710912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343381901342552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4993157510564 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5258806608384 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 170773504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚379531416876032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011601466880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347472647867392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352484267337216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352484267337216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50660878 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199827684 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50660876 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5258659774976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28562197 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347472665870336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343382572819876 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86840724 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010945091072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352483592958464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347472665870336 $ _timestamp‚1.788409521391835e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347472647867392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010945091072ê Ú¨2ž- _wandbruntime‚4624  _runtime‚4624:"\ŸÖ 2loss/SRqqt1/assignment_loss‚2.0143745115091063 /loss/FRt3/detection_loss‚1.7873542754068261 /loss/FRt2/assignment_loss‚3.456277126190709 2loss/SRqqt3/assignment_loss‚0.9445579270913935 2loss/SRqqt4/assignment_loss‚1.5367028742351316 1loss/SRqqt3/detection_loss‚1.0768356152009546 _runtime‚4624.381315358 0loss/FRt4/assignment_loss‚2.8097599966853273 0loss/FBt1/assignment_loss‚0.2286708612300572 1loss/SRqqt4/detection_loss‚1.7087408301505176 /loss/FBt1/detection_loss‚2.8916353285312653 .loss/FBt4/detection_loss‚3.502244326803419 %loss/total_loss‚44.21427327231579 0loss/FRt1/assignment_loss‚2.5747286133144214 /loss/FBt3/detection_loss‚0.8084020326867789 trainer/global_step‚5999 _step‚491 2loss/SRqqt2/assignment_loss‚1.4086837170987714 1loss/FBt2/assignment_loss‚0.27204881235957146 1loss/FBt3/assignment_loss‚0.11389802559827984 0loss/FBt4/assignment_loss‚0.2545750414328217 .loss/FRt1/detection_loss‚3.195132068965746 0loss/FRt3/assignment_loss‚1.4967222753025236 /loss/FRt2/detection_loss‚1.9861003565607416 .loss/FRt4/detection_loss‚2.270777635714587 .loss/FBt2/detection_loss‚3.430940732359886 % _timestamp‚1.7884095213936477e+09 1loss/SRqqt1/detection_loss‚2.5189514077942947 1loss/SRqqt2/detection_loss‚1.9261588800926597 epoch‚4¬COža Û /loss/FRt2/assignment_loss‚3.456277126190709 2loss/SRqqt3/assignment_loss‚0.9445579270913935 2loss/SRqqt4/assignment_loss‚1.5367028742351316 1loss/SRqqt3/detection_loss‚1.0768356152009546 _runtime‚4624.381315358 0loss/FRt4/assignment_loss‚2.8097599966853273 0loss/FBt1/assignment_loss‚0.2286708612300572 1loss/SRqqt4/detection_loss‚1.7087408301505176 /loss/FBt1/detection_loss‚2.8916353285312653 .loss/FBt4/detection_loss‚3.502244326803419 %loss/total_loss‚44.21427327231579 0loss/FRt1/assignment_loss‚2.5747286133144214 /loss/FBt3/detection_loss‚0.8084020326867789 trainer/global_step‚5999 _step‚491 2loss/SRqqt2/assignment_loss‚1.4086837170987714 1loss/FBt2/assignment_loss‚0.27204881235957146 1loss/FBt3/assignment_loss‚0.11389802559827984 0loss/FBt4/assignment_loss‚0.2545750414328217 .loss/FRt1/detection_loss‚3.195132068965746 0loss/FRt3/assignment_loss‚1.4967222753025236 /loss/FRt2/detection_loss‚1.9861003565607416 .loss/FRt4/detection_loss‚2.270777635714587 .loss/FBt2/detection_loss‚3.430940732359886 % _timestamp‚1.7884095213936477e+09 1loss/SRqqt1/detection_loss‚2.5189514077942947 1loss/SRqqt2/detection_loss‚1.9261588800926597 epoch‚4 2loss/SRqqt1/assignment_loss‚2.0143745115091063 /loss/FRt3/detection_loss‚1.7873542754068261ëfÄî’¡‚(:‡ ¼íãԢŢ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.343' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»JÞHf¢‚(:Û ¼íãÔû¦‹  disk.drbd2080.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 22.3896484375) proc.memory.percent‚17.10195541381836 disk.sda.in‚ 0.1953125 network.recv‚ 1.720592e+06* disk./.usagePercent‚16.634701907284803$ disk.nvme0c0n1.out‚ 35094.8515625 disk.dm-0.out‚ 35094.8515625 disk.drbd2265.out‚0 disk.drbd2266.out‚0$ memory_percent‚94.18559074401855% disk./.usageGB‚247.86907958984375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 29.28515625 disk.sda3.out‚ 228.90625 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 228.90625! proc.memory.rssMB‚ 2801.984375 disk.rbd0.in‚0 disk.drbd1716.in‚0" disk.nvme0n1.out‚ 35094.8515625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 29.28515625 disk.drbd2265.in‚0 disk.drbd2266.in‚0 cpu‚25.309165946749797 network.sent‚ 1.3818954e+07 disk.sda2.in‚0 disk.sda.out‚ 228.90625 disk.sda3.in‚ 0.1953125' proc.memory.availableMB‚ 952.6328125p”\‚£‚(:… ËíãÔ•Ä­ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øƒo¤‚(:ä ËíãÔéí‹ * disk./.usagePercent‚16.634803798283432 disk.drbd1716.out‚0# disk.dm-0.out‚35097.05712890625 disk.drbd2265.out‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2801.984375% disk./.usageGB‚247.87059783935547 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 229.4765625! disk.nvme0c0n1.in‚ 29.28515625 proc.cpu.threads‚17& disk.nvme0n1.out‚35097.05712890625 disk.dm-0.in‚ 29.28515625 disk.sda.in‚ 0.1953125) proc.memory.percent‚17.10195541381836 cpu‚25.308387799728262 network.sent‚ 1.3927683e+07 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 22.421875 disk.nvme0n1.in‚ 29.28515625 disk.sda.out‚ 229.4765625( disk.nvme0c0n1.out‚35097.05712890625 disk.dm-1.out‚ 229.4765625 disk.drbd2080.out‚0 disk.sda2.out‚0$ memory_percent‚94.18487548828125 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.availableMB‚952.75 network.recv‚ 1.727589e+06 disk.rbd1.in‚0 disk.drbd2266.in‚0µéÖ¯‘¥‚(:† ÚíãÔ ñª gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚163.487' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²åÛu¦‚(:ê ÚíãÔìÈÆ  network.recv‚ 1.73212e+06 disk.nvme0n1.in‚ 29.28515625 disk.dm-1.out‚ 229.83984375 disk.drbd2265.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 29.28515625! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2801.984375 network.sent‚ 1.3933662e+07* disk./.usagePercent‚16.634903897229318% disk./.usageGB‚247.87208938598633& disk.nvme0n1.out‚35097.68994140625 disk.sda3.in‚ 0.1953125$ memory_percent‚94.18492317199707 disk.drbd1716.out‚0 disk.rbd1.out‚ 22.515625# disk.dm-0.out‚35097.68994140625 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0 disk.sda.out‚ 229.83984375 disk.drbd2266.in‚0) proc.memory.percent‚17.10195541381836 cpu‚25.31582973683217 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 952.7421875 disk.sda1.in‚0( disk.nvme0c0n1.out‚35097.68994140625 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 229.83984375bÐê’§‚(:‡ éíãÔå“ gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚184.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚£ Lr¨‚(:ç éíãÔçõÏ  disk.rbd1.out‚ 22.578125! disk.dm-0.out‚35099.595703125 disk.sda2.out‚0 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 network.sent‚ 1.3940461e+07 disk.dm-1.out‚ 230.43359375 disk.sda.out‚ 230.43359375) proc.memory.percent‚17.10195541381836 cpu‚25.331081288902464 network.recv‚ 1.736425e+06 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 230.43359375 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚35099.595703125! proc.memory.rssMB‚ 2801.984375% disk./.usageGB‚247.87237167358398* disk./.usagePercent‚16.634922841786853 disk.rbd0.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.dm-0.in‚ 29.28515625 disk.drbd2265.out‚0( proc.memory.availableMB‚ 952.70703125 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 29.28515625 disk.drbd2265.in‚0$ memory_percent‚94.18513774871826 disk.drbd2080.in‚0& disk.nvme0c0n1.out‚35099.595703125`Hþ˜©‚(:’ øíãÔ¢âò gpu.0.gpu‚50.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.840600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅS|5aª‚(:Ö øíãÔÏ¿÷  disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2801.984375 proc.cpu.threads‚17* disk./.usagePercent‚16.634968155120415& disk.nvme0c0n1.out‚35100.455078125 disk.nvme0n1.in‚ 29.28515625 disk.sda3.out‚ 230.90625 disk.sda.in‚ 0.1953125 network.sent‚ 1.3944995e+07 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.rbd1.out‚ 22.578125 disk.drbd2080.in‚0$ memory_percent‚94.18671131134033) proc.memory.percent‚17.10195541381836 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 cpu‚25.3463809116852 disk.sda2.out‚0! disk.dm-0.out‚35100.455078125 disk.sda.out‚ 230.90625 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 network.recv‚ 1.74025e+06 disk.sda2.in‚0 disk.drbd1716.in‚0$ disk.nvme0n1.out‚35100.455078125( proc.memory.availableMB‚ 952.44921875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 29.28515625 disk.drbd2265.out‚0 disk./.usageGB‚ 247.873046875 disk.drbd1716.out‚0 disk.dm-1.out‚ 230.90625 disk.rbd0.out‚0…¸×2«- _wandbruntime‚4705  _runtime‚4705ÝOEgr ¬ÁT VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 $ _timestamp‚1.788409602452152e+09 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 $ _timestamp‚1.788409602452152e+09 F2®- _wandbruntime‚4706  _runtime‚4706efˆDQ)¯ËR QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5066757024256 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚51353638 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚389515283074048 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5315939806208 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚28996322 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 114188230 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚356916811082752 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚88011746 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚342624772119552 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚356916300249600 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚384199462806016 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚88011744 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 114188230 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚351850036055552 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚356916811082752 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5066264194048 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5047612435772 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚4706.362826454 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 143478272 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202215953 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚342624789158912 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5048103350164 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5066264194048 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚351850054058496 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚80349799 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚347672384555324 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 202199974 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚356916300249600 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 119538176 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 $ _timestamp‚1.788409603375023e+09 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 114204207 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5066757024256 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚80349960 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚351850054058496 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚351850036055552 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚161 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚159lB&9 °ÐR ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5066757024256 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚51353638 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚389515283074048 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5315939806208 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚28996322 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 114188230 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚356916811082752 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚88011746 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚342624772119552 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚356916300249600 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚384199462806016 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚88011744 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 114188230 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_byte !Nks.large_pool.freed‚351850036055552 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚356916811082752 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5066264194048 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5047612435772 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚4706.362826454 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 143478272 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202215953 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚342624789158912 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5048103350164 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5066264194048 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚351850054058496 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚80349799 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚347672384555324 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 202199974 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚356916300249600 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 119538176 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 $ _timestamp‚1.788409603375023e+09 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 114204207 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5066757024256 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚80349960 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚351850054058496 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚351850036055552 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚161 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚159 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560í}þϱ‚(:’ ‡îãÔ£°û gpu.0.gpu‚22.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.840600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦TD´\²‚(:Ñ ‡îãÔàÄë $ disk./.usageGB‚246.5468635559082 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.out‚ 35102.90625 disk.drbd2265.in‚0 proc.cpu.threads‚17 network.sent‚ 1.3951013e+07 network.recv‚ 1.744723e+06* disk./.usagePercent‚16.545966879834243 disk.drbd2266.out‚0 disk.sda.in‚ 0.1953125! proc.memory.rssMB‚ 2801.984375 disk.drbd2080.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd1716.in‚0 disk.dm-0.in‚ 29.28515625 disk.sda.out‚ 232.046875) proc.memory.percent‚17.10195541381836 disk.dm-1.out‚ 232.046875 disk.drbd2265.out‚0 disk.sda3.out‚ 232.046875 disk.sda2.out‚0& proc.memory.availableMB‚ 953.921875 disk.drbd2266.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.out‚ 35102.90625 disk.dm-1.in‚ 0.1953125 cpu‚25.36155749795252 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 22.7265625 disk.nvme0n1.in‚ 29.28515625 disk.dm-0.out‚ 35102.90625# memory_percent‚94.1777229309082ó÷¡²2³- _wandbruntime‚4719  _runtime‚4719%ßs`•[´é¦ & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004384246189147234 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4908706545829773 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5249596834182739 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_7_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3130652904510498 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.277777761220932 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.266126424074173 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008576680906116962 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚6044 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4903278648853302 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9883354902267456 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4681796133518219 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18661515414714813 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5315934419631958 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5378472208976746 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02318250574171543 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1111111119389534 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 jet/accuracy_3_of_7‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02797987498342991 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1448110044002533 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5561593770980835 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4733290374279022 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19826017320156097 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5439814925193787 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9921388626098633 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 particle/accuracy_4_of_4‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4569060802459717 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004637410864233971 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14752525091171265 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12765957415103912 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943929314613342 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15261635184288025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17881309986114502 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393940567970276 particle/accuracy_5_of_6‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1342625766992569 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.005613330285996199 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5636196732521057 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08154761791229248 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2499696910381317 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5392100811004639 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3535555899143219 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.46971678733825684 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20748119056224823 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09531674534082413 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16733361780643463 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954620599746704 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24750889837741852 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25555557012557983 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18019355833530426 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5229379534721375 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.569444477558136 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 particle/accuracy_1_of_6‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16377170383930206 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13957971334457397 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3268277049064636 jet/accuracy_6_of_6‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013071895577013493 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.39311522245407104 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7083333134651184 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6787294149398804 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08472083508968353 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2076895833015442 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4282567501068115 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6169643402099609 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14643844962120056 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.260553240776062 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3391316533088684 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07371590286493301 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.277777761220932 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31992948055267334 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5427000522613525 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 jet/accuracy_1_of_7‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30351993441581726 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21621622145175934 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5977011322975159 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3680555522441864 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40052372217178345 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49672460556030273 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4716307520866394 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34723272919654846 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1369199901819229 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.975933849811554 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4166666865348816 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4257310628890991 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39635413885116577 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5679011940956116 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4375000298023224 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26461759209632874 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 _step‚494 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_5‚1 jet/accuracy_5_of_6‚0 % _timestamp‚1.7884096160949495e+09 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5122601985931396 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.032977182418107986 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 jet/accuracy_6_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.495650976896286 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5202788710594177 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4891401529312134 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020280888304114342 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6033819913864136 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5620264410972595 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9964697957038879 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.40681344270706177 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8124916553497314 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07364386320114136 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44999998807907104 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.47340065240859985 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862748980522156 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1790679544210434 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.45003220438957214 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15817035734653473 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4917389750480652 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2504991888999939 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9843875169754028 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33796295523643494 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19598767161369324 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980243444442749 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.183389350771904 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38186347484588623 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5750092267990112 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9934807419776917 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.368561714887619 particle/accuracy_1_of_4‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5056341886520386 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014967278577387333 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 )jet/accuracy_1_of_4‚0.914966881275177 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4641057550907135 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24892565608024597 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7521268725395203 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21091528236865997 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5192238092422485 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011992945335805416 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *particle/specificity‚0.946121096611023 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49646198749542236 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12225371599197388 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007741757668554783 particle/accuracy_1_of_7‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.510333240032196 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845407009124756 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9887415170669556 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9985470175743103 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15146198868751526 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06100974231958389 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916204214096069 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21840554475784302 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5140491724014282 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12586775422096252 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4422414302825928 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5255013108253479 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRâþ Û7t1SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_4_of_7‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6355779767036438 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9815667867660522 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1886286586523056 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547619104385376 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1539987176656723 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5430273413658142 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 )loss/val_total_loss‚43.67050968728235 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18484795093536377 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48076337575912476 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 6"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5270956158638 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007275132928043604 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15030156075954437 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4854893088340759 particle/accuracy_2_of_7‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9857961535453796 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5262376070022583 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13252472877502441 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚4719.084428644 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5516641736030579 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1282937228679657 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5124108195304871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11529058963060379 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15106087923049927 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9611239433288574 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9905016422271729 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9787762761116028 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.012544803321361542 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5433087348937988 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15817035734653473 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.013071895577013493 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881068468093872 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.990084171295166 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1569102704524994 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0.5 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24392066895961761 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16001512110233307 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1632183939218521 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3161029517650604 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8678125143051147 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7083333134651184 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2549603283405304 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024539003148674965 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13407528400421143 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.027551021426916122 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3732777535915375 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999630868434906 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2385835349559784 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.11538461595773697 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.567993700504303 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38081395626068115 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 epoch‚4 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 *jet/accuracy_2_of_6‚0.5320513248443604 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904406666755676 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4689542055130005 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9957525134086609 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5100592374801636 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04140562564134598 +jet/accuracy_2_of_2‚0.09956397861242294 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895758628845215 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11989981681108475 particle/accuracy_3_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.237931028008461 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_6_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023694317787885666 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_2‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15111111104488373 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06188080832362175 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31631943583488464 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.22853535413742065 *jet/accuracy_2_of_5‚0.5103110074996948 particle/accuracy_4_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3870701193809509 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08930374681949615 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5424992442131042 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5700103044509888 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5027483701705933 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5319690108299255 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.434929758310318 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 (particle/accuracy‚0.8477595448493958 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7validation_average_jet_accuracy‚0.43357396125793457 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9615384340286255 ,jet/accuracy_4_of_6‚0.057692307978868484 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.39035090804100037 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24459467828273773 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5056341886520386 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5392881631851196 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5335328578948975 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40825220942497253 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4910235106945038 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3299107253551483 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645IluHµî¦ 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5124108195304871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11529058963060379 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15106087923049927 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9611239433288574 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9905016422271729 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9787762761116028 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.012544803321361542 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5433087348937988 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15817035734653473 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.013071895577013493 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881068468093872 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.990084171295166 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1569102704524994 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0.5 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24392066895961761 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16001512110233307 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1632183939218521 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3161029517650604 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8678125143051147 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7083333134651184 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2549603283405304 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024539003148674965 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13407528400421143 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.027551021426916122 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3732777535915375 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999630868434906 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2385835349559784 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.11538461595773697 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.567993700504303 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38081395626068115 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 epoch‚4 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 *jet/accuracy_2_of_6‚0.5320513248443604 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904406666755676 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4689542055130005 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9957525134086609 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5100592374801636 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04140562564134598 +jet/accuracy_2_of_2‚0.09956397861242294 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895758628845215 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11989981681108475 particle/accuracy_3_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.237931028008461 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_6_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023694317787885666 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_2‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15111111104488373 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06188080832362175 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31631943583488464 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.22853535413742065 *jet/accuracy_2_of_5‚0.5103110074996948 particle/accuracy_4_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3870701193809509 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08930374681949615 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5424992442131042 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5700103044509888 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5027483701705933 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5319690108299255 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.434929758310318 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 (particle/accuracy‚0.8477595448493958 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7validation_average_jet_accuracy‚0.43357396125793457 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9615384340286255 ,jet/accuracy_4_of_6‚0.057692307978868484 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.39035090804100037 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24459467828273773 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5056341886520386 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5392881631851196 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5335328578948975 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40825220942497253 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4910235106945038 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3299107253551483 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004384246189147234 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4908706545829773 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5249596834182739 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_7_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3130652904510498 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.277777761220932 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.266126424074173 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008576680906116962 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚6044 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4903278648853302 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9883354902267456 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4681796133518219 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18661515414714813 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5315934419631958 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5378472208976746 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02318250574171543 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1111111119389534 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 jet/accuracy_3_of_7‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02797987498342991 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1448110044002533 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5561593770980835 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4733290374279022 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19826017320156097 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5439814925193787 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9921388626098633 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 particle/accuracy_4_of_4‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4569060802459717 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004637410864233971 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14752525091171265 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12765957415103912 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943929314613342 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15261635184288025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17881309986114502 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393940567970276 particle/accuracy_5_of_6‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1342625766992569 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.005613330285996199 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5636196732521057 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08154761791229248 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2499696910381317 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5392100811004639 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3535555899143219 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.46971678733825684 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20748119056224823 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09531674534082413 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16733361780643463 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954620599746704 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24750889837741852 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25555557012557983 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789kR"È^K738714695 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18019355833530426 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5229379534721375 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.569444477558136 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 particle/accuracy_1_of_6‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16377170383930206 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13957971334457397 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3268277049064636 jet/accuracy_6_of_6‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013071895577013493 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.39311522245407104 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7083333134651184 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6787294149398804 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08472083508968353 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2076895833015442 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4282567501068115 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6169643402099609 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14643844962120056 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.260553240776062 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3391316533088684 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07371590286493301 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.277777761220932 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31992948055267334 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5427000522613525 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 jet/accuracy_1_of_7‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30351993441581726 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21621622145175934 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5977011322975159 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3680555522441864 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40052372217178345 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49672460556030273 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4716307520866394 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34723272919654846 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1369199901819229 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.975933849811554 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4166666865348816 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4257310628890991 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39635413885116577 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5679011940956116 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4375000298023224 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26461759209632874 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 _step‚494 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_5‚1 jet/accuracy_5_of_6‚0 % _timestamp‚1.7884096160949495e+09 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5122601985931396 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.032977182418107986 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 jet/accuracy_6_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.495650976896286 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5202788710594177 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4891401529312134 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020280888304114342 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6033819913864136 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5620264410972595 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9964697957038879 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.40681344270706177 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8124916553497314 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07364386320114136 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44999998807907104 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.47340065240859985 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862748980522156 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1790679544210434 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.45003220438957214 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15817035734653473 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4917389750480652 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2504991888999939 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9843875169754028 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33796295523643494 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19598767161369324 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980243444442749 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.183389350771904 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38186347484588623 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5750092267990112 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9934807419776917 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.368561714887619 particle/accuracy_1_of_4‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5056341886520386 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014967278577387333 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 )jet/accuracy_1_of_4‚0.914966881275177 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4641057550907135 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24892565608024597 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7521268725395203 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21091528236865997 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5192238092422485 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011992945335805416 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *particle/specificity‚0.946121096611023 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49646198749542236 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12225371599197388 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007741757668554783 particle/accuracy_1_of_7‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.510333240032196 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845407009124756 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9887415170669556 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9985470175743103 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15146198868751526 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06100974231958389 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916204214096069 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21840554475784302 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5140491724014282 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12586775422096252 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4422414302825928 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5255013108253479 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_4_of_7‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6355779767036438 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9815667867660522 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1886286586523056 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547619104385376 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1539987176656723 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5430273413658142 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 )loss/val_total_loss‚43.67050968728235 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18484795093536377 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48076337575912476 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 6"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5270956158638 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007275132928043604 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15030156075954437 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4854893088340759 particle/accuracy_2_of_7‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9857961535453796 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5262376070022583 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13252472877502441 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚4719.084428644 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5516641736030579 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1282937228679657î•]A’¶‚(:‡ –îãÔ½²× gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚175.147' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0294 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Èó»j·‚(:ß –îãÔÖèž  disk.nvme0n1.in‚ 29.28515625 disk.dm-0.in‚ 29.28515625 disk.dm-1.out‚ 232.515625 proc.cpu.threads‚17 network.sent‚ 1.4059239e+07" disk.nvme0n1.out‚ 35576.7421875 disk.drbd2266.out‚0 disk.drbd2080.in‚0! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.out‚0$ memory_percent‚91.28496646881104 network.recv‚ 1.751825e+06 disk.dm-0.out‚ 35576.7421875 disk.sda.in‚ 0.1953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚17.107343673706055 disk.sda.out‚ 232.515625 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 35576.7421875) proc.memory.availableMB‚ 1427.87109375% disk./.usageGB‚246.54932403564453 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2802.8671875* disk./.usagePercent‚16.546132004693824 disk.rbd1.out‚ 22.8203125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 232.515625 cpu‚25.368305971461115 QÌå2¸- _wandbruntime‚4726  _runtime‚4726ª6nɹà ' lr-AdamW/pg1‚0.00016125484177071197 ' lr-AdamW/pg2‚0.00016125484177071197 trainer/global_step‚6049 % _timestamp‚1.7884096233206513e+09 _runtime‚4726.308121665 _step‚495ò XÛÎºÈ trainer/global_step‚6049 % _timestamp‚1.7884096233206513e+09 _runtime‚4726.308121665 _step‚495 ' lr-AdamW/pg1‚0.00016125484177071197 ' lr-AdamW/pg2‚0.00016125484177071197ï}&U¯2»- _wandbruntime‚4726  _runtime‚4726x¥Nxà'¼ÚO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114376559 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114356521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51470860 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5324101809664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚352554238585344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202584847 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚415 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348358653992262 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114376559 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352554256588288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51470862 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚4726.309074452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5074787801600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384938474232320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357628370024448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚413 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80544450 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29073590 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚6049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88208286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390262754345984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88208288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5056100914882 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352554256588288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88208288 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343303224554496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88208286 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357628370024448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚413 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80544450 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29073590 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚6049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88208286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390262754345984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88208288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5056100914882 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352554256588288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88208288 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343303224554496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88208286 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884096239995768e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390325251739648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80557114 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114395039 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5074950866944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348415046035110 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202597277 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114395039 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324946381824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚6049 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114374975 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29078086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180387840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚4726.987452561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357686216869888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352610591624192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352610609627136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29078518 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357686216869888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88222304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88222302 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056917350054 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88222304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343358111645696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5074950866944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5325126769664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348414374557786 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390325047411712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114374975 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884096239995768e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390325251739648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80557114 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114395039 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5074950866944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348415046035110 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202597277 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114395039 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324946381824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚6049 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114374975 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29078086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180387840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚4726.987452561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357686216869888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352610591624192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352610609627136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29078518 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357686216869888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88222304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88222302 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056917350054 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88222304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343358111645696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5074950866944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5325126769664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348414374557786 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390325047411712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114374975 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115282573 _runtime‚ 4762.16176565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393388004329472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5366993213952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204211699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51879260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360487682138112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88909088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5096281816534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5115117244928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81178378 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360487682138112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355371908530688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29299116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88909088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81177935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29298675 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5115773607424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355371890527744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5115773607424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360487007772672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355371908530688 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204191659 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346048431082496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346048414043136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5096936254290 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88909086 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928ˆäÝ¢â'ÍÜO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5366993213952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204211699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51879260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360487682138112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88909088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5096281816534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5115117244928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81178378 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360487682138112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355371908530688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29299116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88909088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81177935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29298675 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5115773607424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355371890527744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5115773607424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360487007772672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355371908530688 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204191659 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346048431082496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346048414043136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5096936254290 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88909086 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115282573 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355371890527744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393387808376832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351145367336786 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115282573 _runtime‚ 4762.16176565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393388004329472ôïH622Î- _wandbruntime‚4762  _runtime‚4762)mO$*%ÏåM % _timestamp‚1.7884096598522706e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393450305499136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5097098251498 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393450501438464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346103335213056 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355428261569536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88923104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51887430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115301027 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360544854618112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88923102 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115321091 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81191041 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115321091 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355428261569536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388082661853184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346103318173696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88923102 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360544180239360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360544854618112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29303611 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5097752689462 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355428243566592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351200416425194 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5116593048576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5115936672768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81190588 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5115936672768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204224129 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚4762.840219103 _step‚501 H:DeviceStatsMonitor.on_train_batch_end/active.al7Ñ,÷Ál.allocated‚ 204244195 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29303160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351201087902518 trainer/global_step‚6099 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0aûj4ð&ÐêM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360544854618112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88923102 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115321091 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81191041 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115321091 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355428261569536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388082661853184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346103318173696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88923102 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360544180239360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360544854618112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29303611 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5097752689462 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355428243566592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351200416425194 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5116593048576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5115936672768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81190588 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5115936672768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204224129 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚4762.840219103 _step‚501 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204244195 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29303160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351201087902518 trainer/global_step‚6099 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884096598522706e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393450305499136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5097098251498 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393450501438464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346103335213056 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355428261569536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88923104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51887430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115301027 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889õw82Ñ- _wandbruntime‚4762  _runtime‚4762?¦òSYÒÓ epoch‚5 0loss/FRt4/assignment_loss‚2.4237258594210553 1loss/SRqqt2/assignment_loss‚1.807621359588608 1loss/FBt4/assignment_loss‚0.08009813791035082 1loss/SRqqt3/detection_loss‚1.1026672456533677 .loss/FBt4/detection_loss‚2.199145571752028 /loss/FRt2/detection_loss‚2.1028014593399487 /loss/FRt3/detection_loss‚1.6210472483349883 0loss/SRqqt2/detection_loss‚1.691509139679727 .loss/FBt2/detection_loss‚2.170480728149414 trainer/global_step‚6099 % _timestamp‚1.7884096598540506e+09 0loss/FRt3/assignment_loss‚1.1611632406711578 1loss/FBt2/assignment_loss‚0.29380157564508574 /loss/FRt2/assignment_loss‚3.339416122207275 2loss/SRqqt3/assignment_loss‚1.1437156280021994 0loss/FRt1/assignment_loss‚3.6029101075797247 2loss/SRqqt4/assignment_loss‚1.3721595496684313 0loss/SRqqt4/detection_loss‚1.944661882817745 2loss/SRqqt1/assignment_loss‚1.9806856421753765 /loss/FBt3/detection_loss‚0.4705560503784439 /loss/FRt1/detection_loss‚2.8847004339612763 0loss/FBt3/assignment_loss‚0.1516817561217145 -loss/FRt4/detection_loss‚2.46054093685092 &loss/total_loss‚42.561175705034486 _runtime‚4762.841420048 _step‚502 0loss/FBt1/assignment_loss‚0.7767383837407189 1loss/SRqqt1/detection_loss‚2.3375201523303986 .loss/FBt1/detection_loss‚3.441827493054526»yUŸ^ÓØ 1loss/SRqqt1/detection_loss‚2.3375201523303986 .loss/FBt1/detection_loss‚3.441827493054526 epoch‚5 0loss/FRt4/assignment_loss‚2.4237258594210553 1loss/SRqqt2/assignment_loss‚1.807621359588608 1loss/FBt4/assignment_loss‚0.08009813791035082 1loss/SRqqt3/detection_loss‚1.1026672456533677 .loss/FBt4/detection_loss‚2.199145571752028 /loss/FRt2/detection_loss‚2.1028014593399487 /loss/FRt3/detection_loss‚1.6210472483349883 0loss/SRqqt2/detection_loss‚1.691509139679727 .loss/FBt2/detection_loss‚2.170480728149414 trainer/global_step‚6099 % _timestamp‚1.7884096598540506e+09 0loss/FRt3/assignment_loss‚1.1611632406711578 1loss/FBt2/assignment_loss‚0.29380157564508574 /loss/FRt2/assignment_loss‚3.339416122207275 2loss/SRqqt3/assignment_loss‚1.1437156280021994 0loss/FRt1/assignment_loss‚3.6029101075797247 2loss/SRqqt4/assignment_loss‚1.3721595496684313 0loss/SRqqt4/detection_loss‚1.944661882817745 2loss/SRqqt1/assignment_loss‚1.9806856421753765 /loss/FBt3/detection_loss‚0.4705560503784439 /loss/FRt1/detection_loss‚2.8847004339612763 0loss/FBt3/assignment_loss‚0.1516817561217145 -loss/FRt4/detection_loss‚2.46054093685092 &loss/total_loss‚42.561175705034486 _runtime‚4762.841420048 _step‚502 0loss/FBt1/assignment_loss‚0.7767383837407189öNªžÔ‚(:’ ÃîãÔɦð gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.179' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.235800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kC¡­sÕ‚(:è ÃîãÔÙ‡À  disk.dm-0.in‚ 29.28515625 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 234.328125 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚36056.03955078125* disk./.usagePercent‚16.546220583300673 disk.drbd2265.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2802.8671875 network.sent‚ 1.4297602e+07% disk./.usageGB‚246.55064392089844 disk.sda.out‚ 234.328125 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0$ memory_percent‚94.18485164642334* proc.memory.percent‚17.107343673706055 disk.rbd1.out‚23.03125& disk.nvme0n1.out‚36056.03955078125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0# disk.dm-0.out‚36056.03955078125 disk.nvme0n1.in‚ 29.28515625 disk.sda.in‚ 0.1953125 disk.sda1.out‚0 cpu‚25.365170623224873 disk.dm-1.out‚ 234.328125 disk.drbd2266.in‚0 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 29.28515625( proc.memory.availableMB‚ 952.75390625 disk.dm-1.in‚ 0.1953125 network.recv‚ 1.770855e+06 disk.drbd1716.out‚0ùØO’Ö‚(:‡ ÒîãÔì«¿ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚177.876' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Psè”hׂ(:Ý ÒîãÔ×ÿ‡  disk.drbd2266.in‚0* proc.memory.percent‚17.107343673706055 disk.sda2.in‚0 disk.dm-0.in‚ 29.28515625! disk.dm-0.out‚36057.232421875 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 proc.cpu.threads‚17 network.recv‚ 1.777956e+06! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 29.28515625 disk.drbd2266.out‚0( proc.memory.availableMB‚ 952.73828125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚23.125 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 234.859375 disk.rbd0.in‚0" proc.memory.rssMB‚ 2802.8671875$ disk./.usageGB‚246.5511360168457$ disk.nvme0n1.out‚36057.232421875 disk.sda.out‚ 234.859375 disk.sda3.out‚ 234.859375 network.sent‚ 1.4406092e+07 disk.sda2.out‚0& disk.nvme0c0n1.out‚36057.232421875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚94.18494701385498 cpu‚25.36423344675069) disk./.usagePercent‚16.54625360827259 disk.drbd1716.out‚0» öC2Ø- _wandbruntime‚4798  _runtime‚4798?î§ ÇÙÁ & lr-AdamW/pg1‚0.0001573342377701766 & lr-AdamW/pg2‚0.0001573342377701766 trainer/global_step‚6149 % _timestamp‚1.7884096957288063e+09 _runtime‚4798.716251325 _step‚503ì NZÌÚÆ & lr-AdamW/pg2‚0.0001573342377701766 trainer/global_step‚6149 % _timestamp‚1.7884096957288063e+09 _runtime‚4798.716251325 _step‚503 & lr-AdamW/pg1‚0.0001573342377701766÷Ï×o 2Û- _wandbruntime‚4798  _runtime‚4798ôé:á'ÜÛO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205818511 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81812081 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884096957295086e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116228663 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358189542470144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚4798.717186368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚363345645520896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5156103050752 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116208625 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29523977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391103547998720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353930737727078 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5137771593698 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89609886 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89609888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52287662 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5409655682560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89609886 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363345645520896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358189542470144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116228663 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52287660 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89609888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚363346319886336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5409823500800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358189560473088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29524419 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚391103524058624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348793637610496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5137117155942 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358189560473088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81811637 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚363345645520896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5156103050752 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116208625 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JP<Ëch_start/active_bytes.small_pool.freed‚ 5156103050752 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396513179741184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚363346319886336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116208625 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353931409204194 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚6149 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348793620571136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5156759413248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29523977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391103547998720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353930737727078 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5137771593698 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89609886 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89609888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52287662 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5409655682560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89609886 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363345645520896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358189542470144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116228663 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52287660 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89609888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚363346319886336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5409823500800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358189560473088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29524419 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚391103524058624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872øƒE¤Ä2Þ- _wandbruntime‚4799  _runtime‚4799ägñ2ë&ßåM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚505 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89623904 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52295828 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358245895508992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81824296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4799.400533116 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358245895508992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358245913511936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5156922478592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205850981 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116227079 % _timestamp‚1.7884096964126115e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138588028870 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89623904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353987129769926 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116227079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205871047 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚391165198736384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363403492366336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29528468 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5156922478592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89623902 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5410517284864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205850981 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116247143 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363402817987584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396575692081152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116247143 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391165174796288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52295828 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358245895508992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81824296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4799.400533116 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358245895508992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358245913511936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5156922478592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205850981 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116227079 % _timestamp‚1.7884096964126115e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138588028870 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89623904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353987129769926 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116227079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205871047 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚391165198736384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363403492366336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29528468 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5156922478592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89623902 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5410517284864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205850981 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116247143 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363402817987584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396575692081152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116247143 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391165174796288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361007212415488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366204957634560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399638516548608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90310688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90310686 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52696060 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351538827099136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90310686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366204957634560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361007194412544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207445363 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117154715 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29749221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5197745219072 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351538844138496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5452617133568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356716779594486 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚6199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117134677 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366204283269120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5197745219072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399638702015488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356717451071602 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5197088856576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5177952495350 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207465403 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82445715 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29749653 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361007212415488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361007194412544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5452455606784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90310688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82445281 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366204283269120 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5178606933106 _step‚508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5197088856576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414å<˜å'ïßO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207445363 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117154715 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29749221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5197745219072 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351538844138496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5452617133568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356716779594486 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚6199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117134677 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366204283269120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5197745219072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399638702015488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356717451071602 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5197088856576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5177952495350 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207465403 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82445715 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29749653 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361007212415488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361007194412544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5452455606784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90310688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82445281 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366204283269120 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5178606933106 _step‚508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5197088856576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361007212415488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366204957634560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399638516548608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90310688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90310686 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52696060 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351538827099136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90310686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366204957634560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361007194412544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430üò¶à,2ð- _wandbruntime‚4835  _runtime‚4835"*˜Më&ñåM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚4835.828535607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117153131 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351593731229696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207497899 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394247735619584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5197908284416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361063565454336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52704230 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117173195 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117173195 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82458358 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198564660224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207477833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90324704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117153131 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366262130114560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366262130114560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399701031080960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚6199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198564660224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52704228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5453480914944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356772500160010 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197908284416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394247711679488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351593748269056 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚509 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179423368278 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361063547451392 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361063565454336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5453319401472 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356773171637334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82457930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399701216534528 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚509 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179423368278 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361063547451392 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361063565454336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚@ûc‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5453319401472 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356773171637334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82457930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399701216534528 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117173195 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117173195 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82458358 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198564660224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207477833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90324704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117153131 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366262130114560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366262130114560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399701031080960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚6199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198564660224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52704228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5453480914944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356772500160010 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197908284416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394247711679488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351593748269056ý†Áyf2ó- _wandbruntime‚4835  _runtime‚4835§Ø*WôÑ 0loss/FRt3/assignment_loss‚1.8839179926448397 2loss/SRqqt4/assignment_loss‚1.6661241091205738 1loss/FBt1/assignment_loss‚0.22036958392709494 0loss/FBt4/assignment_loss‚0.2308724145934155 trainer/global_step‚6199 2loss/SRqqt1/assignment_loss‚2.2796901445234976 .loss/FBt1/detection_loss‚3.244989722967148 2loss/SRqqt2/assignment_loss‚1.5877973724235879 2loss/SRqqt3/assignment_loss‚0.8472192664951856 /loss/FBt4/detection_loss‚2.5954772450707178 % _timestamp‚1.7884097328422887e+09 /loss/FRt1/detection_loss‚2.8721768567056367 /loss/SRqqt1/detection_loss‚2.51377946330655 0loss/SRqqt4/detection_loss‚1.645007437095046 /loss/FBt2/detection_loss‚2.1837709665298464 _step‚510 .loss/FRt1/assignment_loss‚4.21431996605613 /loss/FRt4/assignment_loss‚2.917915948059248 0loss/FBt2/assignment_loss‚0.1704895001836121 /loss/FRt2/detection_loss‚2.3595211231415827 0loss/FRt2/assignment_loss‚3.0885687078300275 1loss/FBt3/assignment_loss‚0.21632637325387266 .loss/FRt3/detection_loss‚1.640530595680078 1loss/SRqqt3/detection_loss‚1.1982046120382588 epoch‚5 _runtime‚ 4835.82967759 -loss/FRt4/detection_loss‚2.41026032683642 1loss/SRqqt2/detection_loss‚1.7160370322935126 /loss/FBt3/detection_loss‚0.6794591980259411 &loss/total_loss‚44.382825958801824n¦Ö\õÖ /loss/FRt1/detection_loss‚2.8721768567056367 /loss/SRqqt1/detection_loss‚2.51377946330655 0loss/SRqqt4/detection_loss‚1.645007437095046 /loss/FBt2/detection_loss‚2.1837709665298464 _step‚510 .loss/FRt1/assignment_loss‚4.21431996605613 /loss/FRt4/assignment_loss‚2.917915948059248 0loss/FBt2/assignment_loss‚0.1704895001836121 /loss/FRt2/detection_loss‚2.3595211231415827 0loss/FRt2/assignment_loss‚3.0885687078300275 1loss/FBt3/assignment_loss‚0.21632637325387266 .loss/FRt3/detection_loss‚1.640530595680078 1loss/SRqqt3/detection_loss‚1.1982046120382588 epoch‚5 _runtime‚ 4835.82967759 -loss/FRt4/detection_loss‚2.41026032683642 1loss/SRqqt2/detection_loss‚1.7160370322935126 /loss/FBt3/detection_loss‚0.6794591980259411 &loss/total_loss‚44.382825958801824 0loss/FRt3/assignment_loss‚1.8839179926448397 2loss/SRqqt4/assignment_loss‚1.6661241091205738 1loss/FBt1/assignment_loss‚0.22036958392709494 0loss/FBt4/assignment_loss‚0.2308724145934155 trainer/global_step‚6199 2loss/SRqqt1/assignment_loss‚2.2796901445234976 .loss/FBt1/detection_loss‚3.244989722967148 2loss/SRqqt2/assignment_loss‚1.5877973724235879 2loss/SRqqt3/assignment_loss‚0.8472192664951856 /loss/FBt4/detection_loss‚2.5954772450707178 % _timestamp‚1.7884097328422887e+09þÓ~1’ö‚(:‡ ŽïãÔÈÛ± gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.196' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8392 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"@4Dx÷‚(:í ŽïãÔö«ø * disk./.usagePercent‚16.546384940137656 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2802.8671875 disk.rbd1.in‚0% disk.nvme0n1.out‚36063.4072265625 disk.drbd2265.out‚0 disk.sda.out‚ 236.92578125 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚36063.4072265625 disk.rbd1.out‚ 23.4296875 disk.nvme0n1.in‚ 29.28515625 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0$ memory_percent‚94.20058727264404 disk.drbd2266.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17( proc.memory.availableMB‚ 950.17578125 network.sent‚ 1.4532421e+07 network.recv‚ 1.798185e+06% disk./.usageGB‚246.55309295654297 disk.drbd1716.out‚0 disk.dm-0.in‚ 29.28515625 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 236.92578125 disk.sda.in‚ 0.1953125" disk.dm-0.out‚36063.4072265625 disk.sda3.out‚ 236.92578125 disk.drbd2080.out‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 29.28515625* proc.memory.percent‚17.107343673706055 cpu‚25.360260705522126Ï1à ’ø‚(:‡ ïãÔ¶õ¬ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚145.003' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.0006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý™/Òsù‚(:è ïãÔíÚŒ ! disk.nvme0c0n1.in‚ 29.28515625 disk.drbd1716.out‚0 disk.dm-1.out‚ 237.5234375 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0 disk.dm-0.in‚ 29.28515625 cpu‚25.35930633414054 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 237.5234375 disk.rbd1.out‚ 23.4921875% disk.nvme0n1.out‚36065.4970703125" disk.dm-0.out‚36065.4970703125( proc.memory.availableMB‚ 949.90234375 network.recv‚ 1.805818e+06* disk./.usagePercent‚16.546404140702727$ disk./.usageGB‚246.5533790588379 disk.drbd2265.in‚0 disk.sda.out‚ 237.5234375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚36065.4970703125* proc.memory.percent‚17.107343673706055 disk.nvme0n1.in‚ 29.28515625" proc.memory.rssMB‚ 2802.8671875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚94.20225620269775 network.sent‚ 1.4641534e+07 disk.sda1.out‚0tmÓw2ú- _wandbruntime‚4871  _runtime‚4871›`àfÈû ' lr-AdamW/pg2‚0.00015339712553985317 trainer/global_step‚6249 $ _timestamp‚1.788409768172931e+09 _runtime‚4871.160363063 _step‚511 ' lr-AdamW/pg1‚0.000153397125539853171§ýõÍüÇ ' lr-AdamW/pg1‚0.00015339712553985317 ' lr-AdamW/pg2‚0.00015339712553985317 trainer/global_step‚6249 $ _timestamp‚1.788409768172931e+09 _runtime‚4871.160363063 _step‚511ÿ¬¹m2ý- _wandbruntime‚4871  _runtime‚4871)è@á'þÛO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209072215 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118080767 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29974422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5495659559424 _step‚512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91011488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5238074662400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5238074662400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354284033627136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚369063595382784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4871.161469118 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209092255 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369062921017344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359502821461894 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83078882 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359503492939010 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354284050666496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚369063595382784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369062921017344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53104462 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363824846354944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952Aëù_m!ÿàO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91011488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363824846354944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209072215 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118080767 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29974422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5495659559424 _step‚512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91011488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5238074662400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5238074662400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354284033627136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚369063595382784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4871.161469118 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209092255 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369062921017344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359502821461894 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83078882 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359503492939010 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354284050666496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚369063595382784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369062921017344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_seü …éygments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53104462 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5496350288896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369120767862784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118079183 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83091529 trainer/global_step‚6249 _runtime‚4871.844478747 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5238894090240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118099247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884097688566186e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402826598851584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5239550466048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402826792693760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91025502 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91025504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397330272502784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5219604269722 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91025502 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚513 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5239550466048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359558542027418 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354338954797056 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354338937757696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363881199393792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5238894090240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118099247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29978901 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53112630 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5496520190976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53112628 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363881217396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209104685 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚369120093484032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363881199393792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118079183 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397330248562688 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844´§e@ð&‚êM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884097688566186e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402826598851584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5239550466048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402826792693760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91025502 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91025504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397330272502784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5219604269722 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91025502 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚513 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5239550466048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359558542027418 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354338954797056 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354338937757696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363881199393792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5238894090240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118099247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29978901 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53112630 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5496520190976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53112628 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363881217396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209104685 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚369120093484032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363881199393792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118079183 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397330248562688 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91025504 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5496350288896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369120767862784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118079183 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83091529 trainer/global_step‚6249 _runtime‚4871.844478747 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5238894090240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118099247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118986781 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366642498297344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119006819 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362288863329302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366642498297344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5259623174166 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚6299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5279060468224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83712153 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362289534806418 XJDeviceStatsMonitor.on_train_baÄ@ìætch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357029240155136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 4907.524046 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚371922233131008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371921558765568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53512862 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357029257194496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5279716830720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5538649807872 _step‚516 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30199293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118986781 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53512860 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚366642516300288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚371921558765568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30199729 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91712286 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83712591 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5538490378240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5279716830720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 $ _timestamp‚1.788409804536225e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91712288 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119006819 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91712288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚400351158648320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5279060468224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚371922233131008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118986781 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366642498297344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119006819 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362288863329302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366642498297344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5259623174166 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚6299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5279060468224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83712153 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362289534806418 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357029240155136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 4907.524046 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚371922233131008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371921558765568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53512862 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357029257194496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5279716830720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5538649807872 _step‚516 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30199293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118986781 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53512860 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚366642516300288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚371921558765568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30199729 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91712286 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83712591 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5538490378240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5279716830720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 $ _timestamp‚1.788409804536225e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91712288 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119006819 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096„bË¿2’- _wandbruntime‚4908  _runtime‚4908Uvqë&“åM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5279879896064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53521028 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210731537 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371978731232256 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83724750 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83725186 trainer/global_step‚6299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405952322479616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5539513093632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357084161325056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405952130734592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5261094047094 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30204156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚4908.202684835 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30203722 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚400412785445888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91726304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91726304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5280536271872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5539345288704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119005235 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚303 §ø”êM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210731537 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371978731232256 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83724750 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83725186 trainer/global_step‚6299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405952322479616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5539513093632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357084161325056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405952130734592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5261094047094 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30204156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚4908.202684835 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30203722 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚400412785445888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91726304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91726304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5280536271872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5539345288704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119005235 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5279879896064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53521028 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0…ù£¡¥2•- _wandbruntime‚4908  _runtime‚4908(^k‰[–Õ /loss/FRt2/detection_loss‚2.1406294064088747 1loss/SRqqt4/detection_loss‚1.8082594199448216 .loss/FBt4/detection_loss‚2.267917255560557 &loss/total_loss‚43.002164527565846 1loss/FBt3/assignment_loss‚0.07795792324529571 0loss/SRqqt1/detection_loss‚2.583120139508412 _step‚518 1loss/SRqqt2/detection_loss‚1.5590566680981564 _runtime‚4908.204148004 trainer/global_step‚6299 0loss/FRt1/assignment_loss‚3.1680509992267774 0loss/FRt2/assignment_loss‚3.5289877197959205 0loss/FRt4/assignment_loss‚2.8870892133563757 1loss/FBt1/assignment_loss‚0.17800153888917217 .loss/FBt2/detection_loss‚2.694752033267702 $ _timestamp‚1.788409805216462e+09 2loss/SRqqt3/assignment_loss‚1.0746047244454393 2loss/SRqqt4/assignment_loss‚1.3438176083169422 0loss/FBt4/assignment_loss‚0.1761866635400414 /loss/FRt1/detection_loss‚3.3582607067149617 ,loss/FBt1/detection_loss‚2.6536811987559 2loss/SRqqt2/assignment_loss‚1.6958738182599729 1loss/FBt2/assignment_loss‚0.17909119204994436 /loss/FRt4/detection_loss‚2.9311492880806327 1loss/SRqqt3/detection_loss‚1.1558345808069916 /loss/FRt3/assignment_loss‚1.317184625864029 /loss/FRt3/detection_loss‚1.7602121630311012 /loss/FBt3/detection_loss‚0.6771783577351016 epoch‚5 2loss/SRqqt1/assignment_loss‚1.7852672826627205Á×%œ`—Ú /loss/FRt3/assignment_loss‚1.317184625864029 /loss/FRt3/detection_loss‚1.7602121630311012 /loss/FBt3/detection_loss‚0.6771783577351016 epoch‚5 2loss/SRqqt1/assignment_loss‚1.7852672826627205 /loss/FRt2/detection_loss‚2.1406294064088747 1loss/SRqqt4/detection_loss‚1.8082594199448216 .loss/FBt4/detection_loss‚2.267917255560557 &loss/total_loss‚43.002164527565846 1loss/FBt3/assignment_loss‚0.07795792324529571 0loss/SRqqt1/detection_loss‚2.583120139508412 _step‚518 1loss/SRqqt2/detection_loss‚1.5590566680981564 _runtime‚4908.204148004 trainer/global_step‚6299 0loss/FRt1/assignment_loss‚3.1680509992267774 0loss/FRt2/assignment_loss‚3.5289877197959205 0loss/FRt4/assignment_loss‚2.8870892133563757 1loss/FBt1/assignment_loss‚0.17800153888917217 .loss/FBt2/detection_loss‚2.694752033267702 $ _timestamp‚1.788409805216462e+09 2loss/SRqqt3/assignment_loss‚1.0746047244454393 2loss/SRqqt4/assignment_loss‚1.3438176083169422 0loss/FBt4/assignment_loss‚0.1761866635400414 /loss/FRt1/detection_loss‚3.3582607067149617 ,loss/FBt1/detection_loss‚2.6536811987559 2loss/SRqqt2/assignment_loss‚1.6958738182599729 1loss/FBt2/assignment_loss‚0.17909119204994436 /loss/FRt4/detection_loss‚2.9311492880806327 1loss/SRqqt3/detection_loss‚1.1558345808069916†k·¡®’˜‚(:‡ ÙïãÔ¨€Â gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈXýq™‚(:æ ÙïãÔ¬¼”  network.sent‚ 1.476779e+07" disk.dm-0.out‚36074.8701171875 disk.sda1.in‚0$ memory_percent‚94.20366287231445" proc.memory.rssMB‚ 2802.8671875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.recv‚ 1.825865e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 29.296875 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 29.296875& proc.memory.availableMB‚ 949.671875 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.1953125' disk.nvme0c0n1.out‚36074.8701171875% disk./.usageGB‚246.55690002441406% disk.nvme0n1.out‚36074.8701171875 disk.dm-1.out‚ 239.70703125 disk.sda.in‚ 0.1953125 disk.sda.out‚ 239.70703125 disk.rbd0.out‚0 disk.sda2.out‚0 cpu‚25.355502139456824) disk./.usagePercent‚16.54664043565683 disk.sda3.out‚ 239.70703125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 23.6806640625 disk.nvme0n1.in‚ 29.296875 disk.drbd2265.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.1073436737060559DÅÓ‘š‚(:† èïãÔÕÖÄ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.717 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0côZ”z›‚(:ï èïãÔ‘åé  disk.rbd1.out‚ 23.6806640625 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 77.69921875 network.sent‚ 1.4875717e+07$ disk./.usageGB‚246.6032485961914 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 77.69921875 disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚0* proc.memory.percent‚17.107343673706055 disk.sda3.out‚ 240.29296875& proc.memory.availableMB‚ 950.109375 proc.cpu.threads‚17) disk./.usagePercent‚16.54975092719785& disk.nvme0n1.out‚36134.64794921875 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda2.out‚0 network.recv‚ 1.832614e+06# disk.dm-0.out‚36134.64794921875 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.354581801910367 disk.drbd1716.out‚0 disk.dm-0.in‚ 77.69921875 disk.dm-1.out‚ 240.29296875 disk.sda.out‚ 240.29296875 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2802.8671875 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚36134.64794921875$ memory_percent‚94.20099258422852 disk.sda1.in‚0»b+X2œ- _wandbruntime‚4943  _runtime‚4943H:%Éà _step‚519 ' lr-AdamW/pg1‚0.00014944678686190405 ' lr-AdamW/pg2‚0.00014944678686190405 trainer/global_step‚6349 % _timestamp‚1.7884098408757894e+09 _runtime‚4943.863198298J1ËÎžÈ trainer/global_step‚6349 % _timestamp‚1.7884098408757894e+09 _runtime‚4943.863198298 _step‚519 ' lr-AdamW/pg1‚0.00014944678686190405 ' lr-AdamW/pg2‚0.00014944678686190405‡Ï²2Ÿ- _wandbruntime‚4943  _runtime‚4943öë>¢á' ÛO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30425142 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409015103118336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5320702636544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374780196513792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92413086 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5581239768576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119932871 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5320046274048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359774463722496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212325919 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5300458513574 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53921262 % _timestamp‚1.7884098408764617e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374780196513792 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92413086 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53921260 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409014911360000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119912833 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369460168242688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30424696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5581407586816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119932871 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚4943.864112582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 trainer/global_step‚6349 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212345959 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374780870879232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92413088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84345956 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119912833 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365075576673826 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5301112951330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119912833 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369460168242688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30424696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5581407586816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119932871 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚4943.864112582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 trainer/global_step‚6349 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212345959 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374780870879232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92413088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84345956 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119912833 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365075576673826 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5301112951330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5300458513574 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53921262 % _timestamp‚1.7884098408764617e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374780196513792 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ˆ»°oa2¢- _wandbruntime‚4944  _runtime‚4944ºr ©£åM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409077612860416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsM¶; âB"onitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5320865701888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119931287 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5320865701888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4944.542742379 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5301929386502 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5321522077696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84358607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369516521281536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403495346269184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374837368980480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409077429504000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92427102 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53929430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚365130625762234 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884098415548067e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92427102 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84359051 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5582107174912 trainer/global_step‚6349 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212358389 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92427104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119951351 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30429621 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212378455 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119931287 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5582266591232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369516503278592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53929428 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359829350813696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359829367853056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92427104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369516503278592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5321522077696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374838043359232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚365131297239558 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119951351 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844øÂÆð&¤êM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369516503278592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53929428 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359829350813696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359829367853056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92427104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369516503278592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5321522077696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374838043359232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚365131297239558 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119951351 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409077612860416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5320865701888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119931287 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5320865701888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4944.542742379 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5301929386502 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5321522077696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84358607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369516521281536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403495346269184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374837368980480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409077429504000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92427102 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53929430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚365130625762234 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884098415548067e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92427102 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84359051 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5582107174912 trainer/global_step‚6349 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212358389 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92427104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119951351 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30429621 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212378455 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119931287 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5582266591232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430‰@q{2¥- _wandbruntime‚4944  _runtime‚4944åoY¦Ó /loss/FRt3/assignment_loss‚1.338433439201779 .loss/FRt1/detection_loss‚3.409860042234262 -loss/FBt1/detection_loss‚3.48728404045105 _runtime‚4944.544018756 1loss/FBt1/assignment_loss‚0.24076848633485498 /loss/SRqqt1/detection_loss‚2.44763244600857 _step‚522 /loss/FRt1/assignment_loss‚4.104071120421092 0loss/SRqqt2/detection_loss‚1.499960555029767 2loss/SRqqt4/assignment_loss‚1.8471368584974572 1loss/FBt3/assignment_loss‚0.09129708027700453 .loss/FRt2/detection_loss‚2.067172524282488 /loss/FBt3/detection_loss‚0.5278393562572697 &loss/total_loss‚42.637068849060995 % _timestamp‚1.7884098415566373e+09 0loss/FBt4/assignment_loss‚0.2954017475519019 0loss/FRt2/assignment_loss‚3.0897339471315934 0loss/FRt4/assignment_loss‚2.6612946313360464 2loss/SRqqt2/assignment_loss‚1.3662453527961458 epoch‚5 trainer/global_step‚6349 /loss/FBt2/detection_loss‚2.3454454764723778 /loss/FBt4/detection_loss‚1.8923767606417339 1loss/SRqqt4/detection_loss‚1.5886671042134022 2loss/SRqqt1/assignment_loss‚2.2561331794542423 2loss/SRqqt3/assignment_loss‚0.9519089680877241 1loss/FBt2/assignment_loss‚0.27279023855226114 /loss/FRt3/detection_loss‚1.5383774762352307 .loss/FRt4/detection_loss‚2.220909408901049 1loss/SRqqt3/detection_loss‚1.0963286086916924¾ G^§Ø /loss/FRt3/assignment_loss‚1.338433439201779 .loss/FRt1/detection_loss‚3.409860042234262 -loss/FBt1/detection_loss‚3.48728404045105 _runtime‚4944.544018756 1loss/FBt1/assignment_loss‚0.24076848633485498 /loss/SRqqt1/detection_loss‚2.44763244600857 _step‚522 /loss/FRt1/assignment_loss‚4.104071120421092 0loss/SRqqt2/detection_loss‚1.499960555029767 2loss/SRqqt4/assignment_loss‚1.8471368584974572 1loss/FBt3/assignment_loss‚0.09129708027700453 .loss/FRt2/detection_loss‚2.067172524282488 /loss/FBt3/detection_loss‚0.5278393562572697 &loss/total_loss‚42.637068849060995 % _timestamp‚1.7884098415566373e+09 0loss/FBt4/assignment_loss‚0.2954017475519019 0loss/FRt2/assignment_loss‚3.0897339471315934 0loss/FRt4/assignment_loss‚2.6612946313360464 2loss/SRqqt2/assignment_loss‚1.3662453527961458 epoch‚5 trainer/global_step‚6349 /loss/FBt2/detection_loss‚2.3454454764723778 /loss/FBt4/detection_loss‚1.8923767606417339 1loss/SRqqt4/detection_loss‚1.5886671042134022 2loss/SRqqt1/assignment_loss‚2.2561331794542423 2loss/SRqqt3/assignment_loss‚0.9519089680877241 1loss/FBt2/assignment_loss‚0.27279023855226114 /loss/FRt3/detection_loss‚1.5383774762352307 .loss/FRt4/detection_loss‚2.220909408901049 1loss/SRqqt3/detection_loss‚1.0963286086916924ŠÒK᧨‚(:’ ÷ïãÔ¿Ô¨ gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.856' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.971199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘R­sz©‚(:ï ÷ïãÔá‹”  disk.drbd2265.in‚0 disk.sda.out‚ 242.26171875 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 242.26171875 disk.drbd2266.in‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2802.8671875 network.sent‚ 1.4881744e+07$ disk./.usageGB‚246.7099266052246 disk.dm-1.out‚ 242.26171875( disk.nvme0c0n1.out‚36223.79052734375 disk.rbd1.in‚0* disk./.usagePercent‚16.556910177892878 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0$ memory_percent‚94.19512748718262' proc.memory.availableMB‚ 951.0703125 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 77.72265625 disk.drbd1716.in‚0 disk.dm-0.in‚ 77.72265625 disk.dm-1.in‚ 0.1953125 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.107343673706055 disk.rbd0.in‚0 disk.rbd1.out‚ 23.712890625& disk.nvme0n1.out‚36223.79052734375 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 network.recv‚ 1.836556e+06 disk.nvme0n1.in‚ 77.72265625# disk.dm-0.out‚36223.79052734375 disk.rbd0.out‚0 cpu‚25.35365997989769Zz@鑪‚(:† †ðãÔÄߦ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.905' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.781 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ^Ûz«‚(:ï †ðãÔ÷ˆì  disk.drbd2080.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚36328.46240234375' proc.memory.availableMB‚ 951.3203125 disk.drbd2080.out‚0 disk.dm-0.in‚ 77.73046875 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 cpu‚25.352700372176013 proc.cpu.threads‚17 disk.dm-1.out‚ 242.78515625 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚36328.46240234375 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 77.73046875 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 network.recv‚ 1.843746e+06 disk.sda2.in‚0# disk.dm-0.out‚36328.46240234375 disk.sda3.out‚ 242.78515625% disk./.usageGB‚246.81568908691406 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 77.73046875" proc.memory.rssMB‚ 2802.8671875* proc.memory.percent‚17.107343673706055 disk.sda.out‚ 242.78515625) disk./.usagePercent‚16.56400798677969 disk.drbd1716.out‚0 network.sent‚ 1.499023e+07 disk.rbd0.out‚0 disk.rbd1.out‚ 23.806640625 disk.drbd2265.in‚0$ memory_percent‚94.19360160827637£û•Ê‘¬‚(:† •ðãԇť gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.533' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7066 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†`®öw­‚(:ì •ðãÔ´½ì  disk.sda1.in‚0 disk.sda2.out‚0# disk.dm-0.out‚36442.82568359375 disk.drbd2266.out‚0 disk.drbd2080.in‚0$ proc.memory.availableMB‚950.8125" proc.memory.rssMB‚ 2802.8671875& disk.nvme0n1.out‚36442.82568359375* proc.memory.percent‚17.107343673706055 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 243.23046875 disk.sda2.in‚0( disk.nvme0c0n1.out‚36442.82568359375 disk.dm-0.in‚ 77.73828125 disk.sda.out‚ 243.23046875 disk.rbd1.in‚0 network.sent‚ 1.499655e+07* disk./.usagePercent‚16.568020904878907 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0$ memory_percent‚94.19670104980469% disk./.usageGB‚246.87548446655273 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 77.73828125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 23.869140625 disk.dm-1.out‚ 243.23046875 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 77.73828125 cpu‚25.35183272879102 network.recv‚ 1.848238e+06«EB2®- _wandbruntime‚4980  _runtime‚4980b¤±Pɯà ' lr-AdamW/pg1‚0.00014746768625161841 ' lr-AdamW/pg2‚0.00014746768625161841 trainer/global_step‚6399 % _timestamp‚1.7884098774335194e+09 _runtime‚4980.421007966 _step‚5239µ§!Î°È _step‚523 ' lr-AdamW/pg1‚0.00014746768625161841 ' lr-AdamW/pg2‚0.00014746768625161841 trainer/global_step‚6399 % _timestamp‚1.7884098774335194e+09 _runtime‚4980.421007966‹3vwa2±- _wandbruntime‚4980  _runtime‚4980´Ï¯»P²ÛO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84980059 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377638834262016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624164002304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372277802182144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚362519653211136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377639508627456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120858923 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30650397 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120838885 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93113886 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚524 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5624319237632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367860947064118 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5361688442368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54329660 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213952771 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406516232414720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367861618541234 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5341293852982 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.allÎ`‘.allocated‚377639508627456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5361032079872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120838885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372277820185088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412140372476928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚406516208474624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377638834262016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624164002304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372277802182144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚362519653211136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377639508627456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120858923 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30650397 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120838885 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93113886 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚524 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5624319237632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367860947064118 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5361688442368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54329660 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213952771 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406516232414720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367861618541234 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5341293852982 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377639508627456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5361032079872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120838885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372277820185088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412140372476928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚406516208474624 J×ë&µåM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213985241 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5625184180224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372334155220992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93127904 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367917339106966 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84992245 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30654417 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93127904 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚6399 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5625016375296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372334173223936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377696681107456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412203067332608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5342110287946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93127902 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120857339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214005307 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372334155220992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120877403 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377696681107456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54337830 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120857339 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚525 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120877403 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5342764725910 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367916667629642 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884098781193478e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372334173223936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84992690 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54337828 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844`‚xI ¶êM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93127902 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5362507883520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5361851507712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377696681107456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412203067332608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5342110287946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93127902 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120857339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214005307 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372334155220992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120877403 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377696681107456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54337830 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120857339 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚525 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120877403 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5342764725910 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367916667629642 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884098781193478e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372334173223936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84992690 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54337828 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213985241 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5625184180224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372334155220992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93127904 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367917339106966 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84992245 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30654417 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93127904 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚6399 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5625016375296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372334173223936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5402674248192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30876965 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93814686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85614579 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215599663 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5667260854272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5016.793895636 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375095472127488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365264859739136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54738060 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215579623 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370647660408642 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93814688 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380498146375680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93814688 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93814686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121764937 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409598745357824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409598769297920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415266030152192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121784975 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375095454124544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380497472010240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375095454124544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5667090938880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370646988931526 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5402017885696 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 $ _timestamp‚1.788409913806105e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5382129192390 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380497472010240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30876519 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380498146375680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680Ö;=9å'ÃßO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5402674248192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30876965 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93814686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85614579 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215599663 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5667260854272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5016.793895636 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375095472127488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365264859739136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54738060 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215579623 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370647660408642 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93814688 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380498146375680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93814688 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93814686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121764937 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409598745357824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409598769297920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415266030152192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121784975 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375095454124544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380497472010240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375095454124544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5667090938880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370646988931526 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5402017885696 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 $ _timestamp‚1.788409913806105e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5382129192390 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380497472010240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30876519 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380498146375680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365264876778496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54738062 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121784975 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375095472127488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5402837313536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375151807163392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85627226 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884099144917045e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409660420035584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54746228 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5402837313536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚6449 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚380555318855680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375151825166336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93828702 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121783391 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54746230 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85627676 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93828704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409660396095488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215632159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30881446 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121803455 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5668110798336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365319763869696 _step‚529 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370702709497050 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380555318855680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215612093 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365319780909056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30880998 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5382945627354Jkî&ÆèM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5382945627354 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93828702 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121783391 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5402837313536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375151807163392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85627226 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884099144917045e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409660420035584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54746228 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5402837313536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚6449 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚380555318855680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375151825166336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93828702 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121783391 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54746230 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85627676 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93828704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409660396095488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215632159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30881446 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121803455 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5668110798336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365319763869696 _step‚529 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370702709497050 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380555318855680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215612093 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365319780909056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30880998 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430‘Ew'2Ç- _wandbruntime‚5017  _runtime‚5017ä­ïoXÈÒ /loss/FRt4/assignment_loss‚2.789650902799938 2loss/SRqqt3/assignment_loss‚0.9864410492085406 /loss/SRqqt2/detection_loss‚1.88845589527717 1loss/FBt1/assignment_loss‚0.27069253470229643 1loss/FBt4/assignment_loss‚0.22576217646045343 1loss/SRqqt3/detection_loss‚0.8870238852349744 .loss/FBt1/detection_loss‚2.560957590738932 %loss/total_loss‚42.67830164459996 % _timestamp‚1.7884099144936528e+09 2loss/SRqqt1/assignment_loss‚1.7071725235957849 2loss/SRqqt2/assignment_loss‚1.6071137550931711 /loss/FRt3/detection_loss‚1.6606635246425867 .loss/FBt2/detection_loss‚3.037158409754435 epoch‚5 0loss/FRt2/assignment_loss‚3.5599405020475388 2loss/SRqqt4/assignment_loss‚2.0147182969697592 0loss/FBt2/assignment_loss‚0.3791619671198229 0loss/FBt3/assignment_loss‚0.1448586540906662 /loss/FBt3/detection_loss‚0.6018763621037976 0loss/FRt1/assignment_loss‚2.8571107625961303 .loss/FRt1/detection_loss‚2.808246866294316 0loss/SRqqt4/detection_loss‚1.859827488403896 _runtime‚5017.481070557 0loss/FRt3/assignment_loss‚1.2415583844368274 0loss/SRqqt1/detection_loss‚2.345276537694429 .loss/FBt4/detection_loss‚2.480091094970703 /loss/FRt2/detection_loss‚2.2448272444307804 /loss/FRt4/detection_loss‚2.5197152359330137 trainer/global_step‚6449 _step‚530ãHÚj]É× /loss/FRt4/assignment_loss‚2.789650902799938 2loss/SRqqt3/assignment_loss‚0.9864410492085406 /loss/SRqqt2/detection_loss‚1.88845589527717 1loss/FBt1/assignment_loss‚0.27069253470229643 1loss/FBt4/assignment_loss‚0.22576217646045343 1loss/SRqqt3/detection_loss‚0.8870238852349744 .loss/FBt1/detection_loss‚2.560957590738932 %loss/total_loss‚42.67830164459996 % _timestamp‚1.7884099144936528e+09 2loss/SRqqt1/assignment_loss‚1.7071725235957849 2loss/SRqqt2/assignment_loss‚1.6071137550931711 /loss/FRt3/detection_loss‚1.6606635246425867 .loss/FBt2/detection_loss‚3.037158409754435 epoch‚5 0loss/FRt2/assignment_loss‚3.5599405020475388 2loss/SRqqt4/assignment_loss‚2.0147182969697592 0loss/FBt2/assignment_loss‚0.3791619671198229 0loss/FBt3/assignment_loss‚0.1448586540906662 /loss/FBt3/detection_loss‚0.6018763621037976 0loss/FRt1/assignment_loss‚2.8571107625961303 .loss/FRt1/detection_loss‚2.808246866294316 0loss/SRqqt4/detection_loss‚1.859827488403896 _runtime‚5017.481070557 0loss/FRt3/assignment_loss‚1.2415583844368274 0loss/SRqqt1/detection_loss‚2.345276537694429 .loss/FBt4/detection_loss‚2.480091094970703 /loss/FRt2/detection_loss‚2.2448272444307804 /loss/FRt4/detection_loss‚2.5197152359330137 trainer/global_step‚6449 _step‚530’ý×aÊ‚(:’ ÂðãÔㇻ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚185.192' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.038399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆK^–yË‚(:î Âðãԕφ * proc.memory.percent‚17.107343673706055 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚36694.23388671875& disk.nvme0n1.out‚36694.23388671875 disk.drbd2266.in‚0 disk.drbd2266.out‚0# disk.dm-0.out‚36694.23388671875 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 network.recv‚ 1.863463e+06 disk.sda1.out‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 77.73828125 disk.drbd1716.in‚0 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0% proc.memory.availableMB‚ 950.96875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 77.73828125 disk.drbd2265.out‚0 disk.sda3.out‚ 244.96484375$ memory_percent‚94.19574737548828* disk./.usagePercent‚16.579712256952593 disk.rbd1.in‚0 disk.rbd1.out‚ 23.9951171875 disk.dm-1.out‚ 244.96484375 disk.rbd0.out‚0 cpu‚25.34910177773498$ disk./.usageGB‚247.0496940612793 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 77.73828125" proc.memory.rssMB‚ 2802.8671875 network.sent‚ 1.5115687e+07 disk.sda.out‚ 244.96484375B¶Cï‘Ì‚(:† ÑðãÔþ…ô gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.365' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.473 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢¼ mqÍ‚(:æ ÑðãÔÝ›º ! disk.dm-0.out‚36758.458984375 network.recv‚ 1.870673e+06 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 77.73828125$ disk.nvme0n1.out‚36758.458984375$ memory_percent‚94.19574737548828 network.sent‚ 1.5224699e+07 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0* disk./.usagePercent‚16.583587699005918 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 77.73828125 disk.dm-1.out‚ 245.4765625 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.1953125 disk.drbd1716.in‚0 disk.rbd1.out‚ 24.1435546875 disk.sda2.in‚0& disk.nvme0c0n1.out‚36758.458984375 disk.drbd2080.out‚0% proc.memory.availableMB‚ 950.96875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda.out‚ 245.4765625 disk.sda3.out‚ 245.4765625* proc.memory.percent‚17.107343673706055 disk.drbd1716.out‚0 disk.dm-0.in‚ 77.73828125 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2802.8671875 cpu‚25.34821195837034% disk./.usageGB‚247.10744094848633 disk.dm-1.in‚ 0.1953125·Û2Î- _wandbruntime‚5053  _runtime‚5053—zbÈÏ ' lr-AdamW/pg1‚0.00014350368461047707 ' lr-AdamW/pg2‚0.00014350368461047707 trainer/global_step‚6499 $ _timestamp‚1.788409950608383e+09 _runtime‚5053.595940333 _step‚531H¸ãÍÐÇ _step‚531 ' lr-AdamW/pg1‚0.00014350368461047707 ' lr-AdamW/pg2‚0.00014350368461047707 trainer/global_step‚6499 $ _timestamp‚1.788409950608383e+09 _runtime‚5053.595940333“/«¿-2Ñ- _wandbruntime‚5053  _runtime‚5053’Üyà'ÒÚO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217206475 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412681306181120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122690989 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122711027 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373433030798934 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418391525868544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31104343 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373433702276050 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377913106066944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚383356784123904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383356109758464 $ _timestamp‚1.788409950609191e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377913124069888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86250805 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383356109758464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94515488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55146462 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94515488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5443660054016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377913124069888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55146460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚5053.596903158 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5443003691520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5423618969554 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31103877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5710219687424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚368010083306496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383356784123904 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94515486 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5443660054016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217226515 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418391325721600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5422964531798 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116Ÿp \áÓßO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_startoDsI'/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚5053.596903158 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5443003691520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5423618969554 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31103877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5710219687424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚368010083306496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383356784123904 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94515486 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5443660054016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217226515 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418391325721600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5422964531798 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217206475 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412681306181120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122690989 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122711027 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373433030798934 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418391525868544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31104343 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373433702276050 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377913106066944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚383356784123904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383356109758464 $ _timestamp‚1.788409950609191e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377913124069888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86250805 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383356109758464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94515488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55146462 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94515488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5443660054016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377913124069888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55146460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872”ƒ42Ô- _wandbruntime‚5054  _runtime‚5054ü„þë&ÕåM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31108971 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94529504 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122709443 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5444479495168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚383413282225152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55154628 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86263151 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418453853945856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412742932978688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368064987437056 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412742956918784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418454033107968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6499 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217259011 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94529502 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377969459105792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373489422841782 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5423780966762 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5424435404726 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5443823119360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373488751364458 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚368064970397696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94529502 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31108523 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401,ëð&ÖêM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412742956918784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418454033107968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6499 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217259011 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94529502 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377969459105792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373489422841782 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5423780966762 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5424435404726 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5443823119360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373488751364458 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚368064970397696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94529502 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31108523 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31108971 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94529504 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122709443 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5444479495168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚383413282225152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55154628 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86263151 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418453853945856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412742932978688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368064987437056 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944•÷v­Ç2×- _wandbruntime‚5054  _runtime‚5054ýÚG\ØÖ .loss/FRt2/detection_loss‚2.400219056256333 .loss/FRt4/detection_loss‚2.335220998951367 /loss/FBt4/detection_loss‚2.6503060195181103 2loss/SRqqt3/assignment_loss‚0.9776638801849913 2loss/SRqqt4/assignment_loss‚1.5788257711101323 _step‚534 1loss/SRqqt1/detection_loss‚2.3079065765653337 /loss/FBt1/assignment_loss‚0.649698394949415 1loss/FBt4/assignment_loss‚0.03961450564222307 /loss/FBt1/detection_loss‚3.1073281509535655 0loss/FRt1/assignment_loss‚3.8282143663276327 2loss/SRqqt1/assignment_loss‚1.9309799943651471 1loss/FBt2/assignment_loss‚0.11026067137717063 /loss/FRt3/detection_loss‚2.0739595292136075 epoch‚5 /loss/FRt3/assignment_loss‚1.130504634976387 2loss/SRqqt2/assignment_loss‚1.6340892953532082 1loss/SRqqt4/detection_loss‚1.8038513335798467 .loss/FBt2/detection_loss‚2.869969779253006 /loss/FBt3/detection_loss‚0.6104485195087777 %loss/total_loss‚44.75424437258303 0loss/FRt4/assignment_loss‚2.5501534314382646 1loss/FBt3/assignment_loss‚0.11599833982597135 .loss/FRt1/detection_loss‚3.177925245328383 1loss/SRqqt2/detection_loss‚2.0567372121981213 1loss/SRqqt3/detection_loss‚1.0869829579023644 % _timestamp‚1.7884099512996361e+09 _runtime‚5054.287048201 0loss/FRt2/assignment_loss‚3.7273857078036747 trainer/global_step‚6499GJL0ÙÛ .loss/FRt4/detection_loss‚2.335220998951367 /loss/FBt4/detection_loss‚2.6503060195181103 2loss/SRqqt3/assignment_loss‚0.9776638801849913 2loss/SRqqt4/assignment_loss‚1.5788257711101323 _step‚534 1loss/SRqqt1/detection_loss‚2.3079065765653337 /loss/FBt1/assignment_loss‚0.649698394949415 1loss/FBt4/assignment_loss‚0.03961450564222307 /loss/FBt1/detection_loss‚3.1073281509535655 0loss/FRt1/assignment_loss‚3.8282143663276327 2loss/SRqqt1/assignment_loss‚1.9309799943651471 1loss/FBt2/assignment_loss‚0.11026067137717063 /loss/FRt3/detection_loss‚2.0739595292136075 epoch‚5 /loss/FRt3/assignment_loss‚1.130504634976387 2loss/SRqqt2/assignment_loss‚1.6340892953532082 1loss/SRqqt4/detection_loss‚1.8038513335798467 .loss/FBt2/detection_loss‚2.869969779253006 /loss/FBt3/detection_loss‚0.6104485195087777 %loss/total_loss‚44.75424437258303 0loss/FRt4/assignment_loss‚2.5501534314382646 1loss/FBt3/assignment_loss‚0.11599833982597135 .loss/FRt1/detection_loss‚3.177925245328383 1loss/SRqqt2/detection_loss‚2.0567372121981213 1loss/SRqqt3/detection_loss‚1.0869829579023644 % _timestamp‚1.7884099512996361e+09 _runtime‚5054.287048201 0loss/FRt2/assignment_loss‚3.7273857078036747 trainer/global_step‚6499 . ÿòÂ1loss/FRt2/detection_loss‚2.400219056256333–ë®ÊÚ‚(:’ àðãÔâ›ç gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.111599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁYJ9qÛ‚(:æ àðãÔ±îí  disk.dm-0.in‚ 77.73828125 disk.drbd2265.out‚0( proc.memory.availableMB‚ 951.17578125 disk.rbd1.out‚ 24.1435546875 proc.cpu.threads‚17 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2802.8671875! disk.nvme0c0n1.in‚ 77.73828125 disk.rbd1.in‚0 disk.rbd0.out‚0 network.sent‚ 1.5230035e+07$ disk./.usageGB‚247.1584243774414 disk.sda.out‚ 246.046875 disk.sda3.out‚ 246.046875 disk.drbd2266.in‚0 network.recv‚ 1.874696e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0! disk.dm-0.out‚36874.919921875 disk.dm-1.in‚ 0.1953125 disk.drbd2080.in‚0$ disk.nvme0n1.out‚36874.919921875 disk.dm-1.out‚ 246.046875 disk.drbd2265.in‚0 cpu‚25.347288267581668 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0$ memory_percent‚94.19448375701904* proc.memory.percent‚17.107343673706055* disk./.usagePercent‚16.587009239701043 disk.sda.in‚ 0.1953125& disk.nvme0c0n1.out‚36874.919921875 disk.nvme0n1.in‚ 77.73828125qd’UÜ‚(:’ ïðãÔ×¹š gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.951' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.590199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï E|Ý‚(:ñ ïðãÔ×È÷  disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 24.17578125 disk.nvme0n1.in‚ 77.73828125 disk.sda.in‚ 0.1953125 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚36948.67529296875 network.recv‚ 1.881633e+06 disk.drbd2266.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 77.73828125( disk.nvme0c0n1.out‚36948.67529296875& disk.nvme0n1.out‚36948.67529296875 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 246.53515625% disk./.usageGB‚247.20916748046875 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2802.8671875 cpu‚25.346401977268776 network.sent‚ 1.5338661e+07* disk./.usagePercent‚16.590414651921506 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 246.53515625 disk.rbd1.in‚0$ memory_percent‚94.20101642608643 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.dm-0.in‚ 77.73828125 disk.drbd2265.out‚0 disk.sda.out‚ 246.53515625 disk.sda3.in‚ 0.1953125( proc.memory.availableMB‚ 950.10546875* proc.memory.percent‚17.107343673706055,Xr’Þ‚(:‡ þðãԛⵠgpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.453' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6906 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0??pÁyß‚(:î þðãÔ‰÷‚  disk.dm-1.in‚ 0.1953125 network.recv‚ 1.886021e+06% disk./.usageGB‚247.26726150512695 disk.drbd1716.in‚0 disk.sda3.out‚ 247.23046875 disk.drbd2266.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚37008.39697265625 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 77.73828125" proc.memory.rssMB‚ 2802.8671875 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 24.35546875 disk.sda.out‚ 247.23046875 disk.sda3.in‚ 0.1953125& disk.nvme0n1.out‚37008.39697265625* proc.memory.percent‚17.107343673706055) disk./.usagePercent‚16.59431339066045$ memory_percent‚94.19951438903809 cpu‚25.34549233280941 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚37008.39697265625' proc.memory.availableMB‚ 950.3515625 network.sent‚ 1.5344643e+07 disk.drbd1716.out‚0 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 77.73828125 disk.dm-1.out‚ 247.23046875 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 77.73828125K¸H}2à- _wandbruntime‚5090  _runtime‚5090²·µ"Éáà ' lr-AdamW/pg1‚0.00014151960967162194 ' lr-AdamW/pg2‚0.00014151960967162194 trainer/global_step‚6549 % _timestamp‚1.7884099870956104e+09 _runtime‚5090.083108582 _step‚535†‚ÎâÈ ' lr-AdamW/pg2‚0.00014151960967162194 trainer/global_step‚6549 % _timestamp‚1.7884099870956104e+09 _runtime‚5090.083108582 _step‚535 ' lr-AdamW/pg1‚0.00014151960967162194—?ÏÑ72ã- _wandbruntime‚5090  _runtime‚5090F»>Vá'äÛO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218833327 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380730776012288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86885473 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421516783155200 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415763843064320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370755272795136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5090.084045167 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95216286 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5484645859840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386215421872128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386214747506688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386214747506688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5483989497344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380730776012288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386215421872128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415763819124224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31330611 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380730758009344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86885034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376219072666342 trainer/global_step‚6549 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370755272795136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5090.084045167 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95216286 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5484645859840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386215421872128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386214747506688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386214747506688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5483989497344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380730776012288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386215421872128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415763819124224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31330611 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380730758009344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86885034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376219072666342 trainer/global_step‚6549 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0˜£Vzè2æ- _wandbruntime‚5090  _runtime‚5090k­ÇÂ@çåM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95230304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31334616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380787129051136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370810176925696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86898079 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚6549 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380787111048192 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386272594352128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218865797 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218865797 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 % _timestamp‚1.7884099877831693e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380787129051136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5465270744134 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386272594352128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421579298174976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386271919973376 _runtime‚5090.771032121 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370810193965056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86897644 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5485465300992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421579485725696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95230304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5753828313088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123655559 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123655559 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370810193965056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86897644 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5485465300992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421579485725696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95230304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5753828313088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123655559 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123655559 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386272594352128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218865797 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218865797 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 % _timestamp‚1.7884099877831693e+09™Fú÷S2é- _wandbruntime‚5090  _runtime‚5090T§ûCZêÔ 1loss/FBt4/assignment_loss‚0.08050388336232572 .loss/FRt1/detection_loss‚2.978444755077362 /loss/FRt2/detection_loss‚2.4254775928414385 1loss/SRqqt3/detection_loss‚1.0313068000508137 /loss/FBt4/detection_loss‚2.1079433150589466 0loss/FRt2/assignment_loss‚2.9803270373655404 1loss/SRqqt4/assignment_loss‚1.473398233548953 1loss/FBt2/assignment_loss‚0.10796048093030615 % _timestamp‚1.7884099877849648e+09 _runtime‚5090.772352451 trainer/global_step‚6549 .loss/FRt1/assignment_loss‚3.53208190202713 1loss/SRqqt2/assignment_loss‚1.718083080947399 1loss/SRqqt1/detection_loss‚2.2878910593084387 1loss/SRqqt4/detection_loss‚1.9731445564673498 2loss/SRqqt1/assignment_loss‚1.8342425015729826 $loss/total_loss‚42.7505115976914 0loss/FBt3/detection_loss‚0.48107246186670016 _step‚538 1loss/SRqqt2/detection_loss‚1.8600988101959228 .loss/FBt2/detection_loss‚3.269150401864733 epoch‚5 0loss/FRt4/assignment_loss‚3.7301521508061155 1loss/SRqqt3/assignment_loss‚0.759634515957632 /loss/FRt3/detection_loss‚1.4811931023339056 .loss/FBt1/detection_loss‚2.361485108733177 0loss/FRt3/assignment_loss‚1.5867801263647259 1loss/FBt1/assignment_loss‚0.34178613785964745 .loss/FRt4/detection_loss‚2.265170456195364 1loss/FBt3/assignment_loss‚0.08318312695448436’ƒÂ_ëÙ 1loss/SRqqt3/assignment_loss‚0.759634515957632 /loss/FRt3/detection_loss‚1.4811931023339056 .loss/FBt1/detection_loss‚2.361485108733177 0loss/FRt3/assignment_loss‚1.5867801263647259 1loss/FBt1/assignment_loss‚0.34178613785964745 .loss/FRt4/detection_loss‚2.265170456195364 1loss/FBt3/assignment_loss‚0.08318312695448436 1loss/FBt4/assignment_loss‚0.08050388336232572 .loss/FRt1/detection_loss‚2.978444755077362 /loss/FRt2/detection_loss‚2.4254775928414385 1loss/SRqqt3/detection_loss‚1.0313068000508137 /loss/FBt4/detection_loss‚2.1079433150589466 0loss/FRt2/assignment_loss‚2.9803270373655404 1loss/SRqqt4/assignment_loss‚1.473398233548953 1loss/FBt2/assignment_loss‚0.10796048093030615 % _timestamp‚1.7884099877849648e+09 _runtime‚5090.772352451 trainer/global_step‚6549 .loss/FRt1/assignment_loss‚3.53208190202713 1loss/SRqqt2/assignment_loss‚1.718083080947399 1loss/SRqqt1/detection_loss‚2.2878910593084387 1loss/SRqqt4/detection_loss‚1.9731445564673498 2loss/SRqqt1/assignment_loss‚1.8342425015729826 $loss/total_loss‚42.7505115976914 0loss/FBt3/detection_loss‚0.48107246186670016 _step‚538 1loss/SRqqt2/detection_loss‚1.8600988101959228 .loss/FBt2/detection_loss‚3.269150401864733 epoch‚5 0loss/FRt4/assignment_loss‚3.7301521508061155š ¹ôaì‚(:’ ñãÔ½¾ï gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.550200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´\ûêwí‚(:ì ñãÔ”³Œ ' disk.nvme0c0n1.out‚37123.2744140625 disk.drbd1716.out‚0% disk.nvme0n1.out‚37123.2744140625" disk.dm-0.out‚37123.2744140625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚16.597665041298725 disk.sda2.out‚0$ memory_percent‚94.20123100280762* proc.memory.percent‚17.107343673706055 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0' proc.memory.availableMB‚ 950.0703125" proc.memory.rssMB‚ 2802.8671875 cpu‚25.34468584624643 disk.dm-1.out‚ 247.89453125 disk.drbd2080.in‚0 disk.rbd1.out‚ 24.41796875 disk.sda.out‚ 247.89453125 disk.nvme0n1.in‚ 77.73828125 network.sent‚ 1.5350954e+07% disk./.usageGB‚247.31720352172852! disk.nvme0c0n1.in‚ 77.73828125 network.recv‚ 1.890629e+06 disk.dm-0.in‚ 77.73828125 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 247.89453125 disk.drbd2080.out‚0öF`î‚(:’ œñãÔÏø„ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.024' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.004799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºhzxvï‚(:ë œñãÔËÔ   disk.drbd2265.in‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2802.8671875 cpu‚25.343775581615507 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1953125$ memory_percent‚94.19972896575928( proc.memory.availableMB‚ 950.31640625 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 24.50390625% disk.nvme0n1.out‚37167.8291015625 disk.sda1.in‚0' disk.nvme0c0n1.out‚37167.8291015625 disk.sda3.in‚ 0.1953125 network.sent‚ 1.5459081e+07 network.recv‚ 1.897568e+06 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 248.3671875* disk./.usagePercent‚16.600882031973992 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 77.73828125 disk.nvme0n1.in‚ 77.73828125 disk.dm-1.out‚ 248.3671875 disk.drbd2265.out‚0* proc.memory.percent‚17.107343673706055 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 248.3671875 disk.drbd1716.in‚0" disk.dm-0.out‚37167.8291015625 disk.drbd2266.in‚0% disk./.usageGB‚247.36513900756836 disk.dm-0.in‚ 77.73828125E#2ð- _wandbruntime‚5126  _runtime‚5126MÞnÇñÁ _step‚539 & lr-AdamW/pg1‚0.0001395347032050827 & lr-AdamW/pg2‚0.0001395347032050827 trainer/global_step‚6599 % _timestamp‚1.7884100238648531e+09 _runtime‚5126.8522872049»aÌòÆ & lr-AdamW/pg2‚0.0001395347032050827 trainer/global_step‚6599 % _timestamp‚1.7884100238648531e+09 _runtime‚5126.852287204 _step‚539 & lr-AdamW/pg1‚0.0001395347032050827› wÜi2ó- _wandbruntime‚5126  _runtime‚5126‚È> á'ôÛO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31559101 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95917086 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124563131 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424642459100672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389074059620352 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418846379947520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379005114533750 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373500479323136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5524975303168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124543093 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87522363 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 trainer/global_step‚6599 _runtime‚5126.853140142 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5505289648370 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5524975303168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95917086 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95917088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383548409951744 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚540 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55963262 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389073385254912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383548409951744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5796079153152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5504635210614 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379005786010866 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389074059620352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124543093 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844ÔÞë? õàO % _timestamp‚1.7884100238655055e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220480219 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124563131 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383548427954688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124543093 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31559101 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95917086 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124563131 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424642459100672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389074059620352 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418846379947520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379005114533750 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373500479323136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5524975303168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚K£ÄxÔ 124543093 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87522363 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 trainer/global_step‚6599 _runtime‚5126.853140142 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5505289648370 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5524975303168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95917086 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95917088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383548409951744 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚540 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55963262 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389073385254912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383548409951744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5796079153152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5504635210614 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379005786010866 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389074059620352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚383604780993536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373555400493056 % _timestamp‚1.7884100245529723e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55971430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5505451645578 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5506106083542 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95931102 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220492649 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6599 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87534624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383604780993536 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5526451106816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379060835099274 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389130557721600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124581611 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424704972434944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220512715 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5796941749760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373555383453696 _step‚541 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87535077 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5796776041984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389131232100352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124561547 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚383604762990592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95931102 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31563647 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048Dû-!ð&øêM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6599 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87534624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383604780993536 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5526451106816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379060835099274 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389130557721600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124581611 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424704972434944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220512715 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5796941749760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373555383453696 _step‚541 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87535077 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5796776041984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389131232100352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124561547 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚383604762990592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95931102 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31563647 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚383604780993536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373555400493056 % _timestamp‚1.7884100245529723e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55971430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5505451645578 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5506106083542 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95931102 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220492649 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521¢Iš2ù- _wandbruntime‚5127  _runtime‚5127ÜÚ•8`úÚ 1loss/SRqqt3/assignment_loss‚1.099936480734598 1loss/SRqqt1/detection_loss‚2.3490867578622066 epoch‚5 1loss/FBt2/assignment_loss‚0.11859740759245962 1loss/SRqqt2/detection_loss‚1.8033556853388917 1loss/SRqqt3/detection_loss‚0.9463185058856333 /loss/FBt3/detection_loss‚0.5064153300867857 0loss/FBt4/assignment_loss‚0.1939443972656818 .loss/FRt1/detection_loss‚3.186652066378758 0loss/FRt1/assignment_loss‚2.8670341680789817 0loss/FRt4/assignment_loss‚2.7391126003919863 2loss/SRqqt1/assignment_loss‚1.9647643419377732 2loss/SRqqt4/assignment_loss‚1.2724623645239688 1loss/SRqqt4/detection_loss‚1.7865239692495225 trainer/global_step‚6599 _runtime‚5127.542246214 2loss/SRqqt2/assignment_loss‚1.3538281120889788 1loss/FBt1/assignment_loss‚0.36323737130574457 /loss/FBt2/detection_loss‚3.0076423436403275 0loss/FRt2/assignment_loss‚3.0170991514858447 1loss/FBt3/assignment_loss‚0.10323986810942448 .loss/FRt3/detection_loss‚1.595536786805967 /loss/FRt4/detection_loss‚2.2361283448396945 /loss/FBt4/detection_loss‚3.1666861921548843 % _timestamp‚1.7884100245548778e+09 0loss/FRt3/assignment_loss‚1.4476122773926834 /loss/FRt2/detection_loss‚2.4323626831174097 &loss/total_loss‚41.878265107158214 _step‚542 /loss/FBt1/detection_loss‚2.3206879008900034²»-êeûß _runtime‚5127.542246214 2loss/SRqqt2/assignment_loss‚1.3538281120889788 1loss/FBt1/assignment_loss‚0.36323737130574457 /loss/FBt2/detection_loss‚3.0076423436403275 0loss/FRt2/assignment_loss‚3.0170991514858447 1loss/FBt3/assignment_loss‚0.10323986810942448 .loss/FRt3/detection_loss‚1.595536786805967 /loss/FRt4/detection_loss‚2.2361283448396945 /loss/FBt4/detection_loss‚3.1666861921548843 % _timestamp‚1.7884100245548778e+09 0loss/FRt3/assignment_loss‚1.4476122773926834 /loss/FRt2/detection_loss‚2.4323626831174097 &loss/total_loss‚41.878265107158214 _step‚542 /loss/FBt1/detection_loss‚2.3206879008900034 1loss/SRqqt3/assignment_loss‚1.099936480734598 1loss/SRqqt1/detection_loss‚2.3490867578622066 epoch‚5 1loss/FBt2/assignment_loss‚0.11859740759245962 1loss/SRqqt2/detection_loss‚1.8033556853388917 1loss/SRqqt3/detection_loss‚0.9463185058856333 /loss/FBt3/detection_loss‚0.5064153300867857 0loss/FBt4/assignment_loss‚0.1939443972656818 .loss/FRt1/detection_loss‚3.186652066378758 0loss/FRt1/assignment_loss‚2.8670341680789817 0loss/FRt4/assignment_loss‚2.7391126003919863 2loss/SRqqt1/assignment_loss‚1.9647643419377732 2loss/SRqqt4/assignment_loss‚1.2724623645239688 1loss/SRqqt4/detection_loss‚1.7865239692495225 trainer/global_step‚6599ž"¥Vü‚(:’ «ñãԤŢ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.856' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.971199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚèL[zý‚(:ï «ñãÔÌ£ï  disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 77.74609375# disk.dm-0.out‚37241.01318359375 disk.sda3.out‚ 248.87890625 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2802.8671875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 248.87890625 disk.sda.out‚ 248.87890625% disk./.usageGB‚247.40527725219727 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 24.56640625 disk.sda3.in‚ 0.1953125$ memory_percent‚94.19987201690674 disk.drbd2080.in‚0! disk.nvme0c0n1.in‚ 77.74609375( proc.memory.availableMB‚ 950.29296875 cpu‚25.34290454052527 proc.cpu.threads‚17 disk.sda.in‚ 0.1953125( disk.nvme0c0n1.out‚37241.01318359375 disk.nvme0n1.in‚ 77.74609375 disk.drbd2266.out‚0 network.sent‚ 1.546505e+07 network.recv‚ 1.901432e+06* disk./.usagePercent‚16.603575743249266 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚37241.01318359375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0* proc.memory.percent‚17.107343673706055ÄÆ$œþ‚(:‘ ºñãÔÏö“ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.146' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚36.22919999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06¨g(ÿ‚(:è ºñãÔзø $ disk./.usageGB‚247.4449691772461 disk.drbd2266.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚17.107343673706055 proc.cpu.threads‚17 disk.rbd1.in‚0& disk.nvme0n1.out‚37322.48583984375 disk.sda.out‚ 249.5703125 disk.drbd2266.out‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 77.74609375* disk./.usagePercent‚16.606239501643035 disk.dm-1.in‚ 0.1953125 disk.drbd1716.in‚0 network.recv‚ 1.908599e+06 disk.nvme0n1.in‚ 77.74609375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚37322.48583984375 disk.dm-0.in‚ 77.74609375# disk.dm-0.out‚37322.48583984375 disk.sda.in‚ 0.1953125$ proc.memory.availableMB‚980.3125 disk.rbd1.out(?ÇZK‚ 24.66015625 disk.dm-1.out‚ 249.5703125 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2802.8671875 cpu‚25.34210209311876 network.sent‚ 1.5573715e+07 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 249.5703125 disk.drbd2080.out‚0 disk.sda1.in‚0$ memory_percent‚94.01664733886719¨0‘€‚(:† ÉñãԺɉ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚177.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.517 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,n‰‘r‚(:ç ÉñãÔî¡Ž  disk.sda.in‚ 0.1953125 cpu‚25.341217313311343 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 77.74609375# disk.nvme0n1.out‚37388.58984375 disk.drbd2265.out‚0 disk.rbd0.in‚0* proc.memory.percent‚17.107343673706055 network.sent‚ 1.5580026e+07* disk./.usagePercent‚16.608478799545033% disk.nvme0c0n1.out‚37388.58984375 disk.sda1.in‚0 disk.rbd1.out‚ 24.72265625 disk.nvme0n1.in‚ 77.74609375 disk.dm-0.in‚ 77.74609375 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-0.out‚37388.58984375 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 250.42578125' proc.memory.availableMB‚ 980.2890625 proc.cpu.threads‚17 network.recv‚ 1.913293e+06 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 250.42578125 disk.drbd1716.out‚0 disk.sda.out‚ 250.42578125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0$ memory_percent‚94.01679039001465" proc.memory.rssMB‚ 2802.8671875% disk./.usageGB‚247.47833633422852 disk.drbd2265.in‚0Ò"92‚- _wandbruntime‚5163  _runtime‚5163bC“ȃ ' lr-AdamW/pg1‚0.00013754937886239186 ' lr-AdamW/pg2‚0.00013754937886239186 trainer/global_step‚6649 $ _timestamp‚1.788410060227486e+09 _runtime‚5163.214949409 _step‚543Ã-Ö>Í„Ç ' lr-AdamW/pg1‚0.00013754937886239186 ' lr-AdamW/pg2‚0.00013754937886239186 trainer/global_step‚6649 $ _timestamp‚1.788410060227486e+09 _runtime‚5163.214949409 _step‚543Ÿ)ãS2…- _wandbruntime‚5163  _runtime‚5163A€Ã/á'†ÛO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56371662 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391932023003136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5566617471488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222087031 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386366079897088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391932023003136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222107071 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421928916830720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125489183 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125469145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31786196 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421928892890624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚381791827878274 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391932697368576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96617888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚5163.216023715 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884100602282088e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚427767763441664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381791156401158 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376245685851136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96617886 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚317857607q[æ'‡àO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56371662 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391932023003136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5566617471488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222087031 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386366079897088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391932023003136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222107071 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421928916830720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125489183 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125469145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31786196 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421928892890624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚381791827878274 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391932697368576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96617888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚5163.216023715 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884100602282088e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚427767763441664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381791156401158 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376245685851136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96617886 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31785760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 «b2ˆ- _wandbruntime‚5163  _runtime‚5163êدs&‰åM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546941422950 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421990543628288 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125507663 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125487599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386422432935936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386422432935936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5839896762368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚5163.891031487 _step‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546286984986 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5839735248896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88170491 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376300607021056 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391989869848576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56379828 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31790661 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚6649 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5566780536832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5566780536832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56379830 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96631904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚427830464330752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚427830278877184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376300589981696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88170066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222139567 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884100609031043e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚391989195469824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96631904 I@DeviceStatsMonitor.on_train_batch_î™E*xend/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560fNû)ð&ŠêM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5567436912640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5567436912640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884100609031043e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚391989195469824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96631904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546941422950 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421990543628288 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125507663 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125487599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386422432935936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386422432935936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5839896762368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚5163.891031487 _step‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546286984986 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5839735248896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88170491 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376300607021056 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391989869848576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56379828 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31790661 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚6649 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5566780536832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5566780536832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56379830 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96631904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚427830464330752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚427830278877184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376300589981696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88170066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222139567 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394790660751360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430893355934720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430893547693056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378990909418496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126415235 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚394791335116800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5586960327186 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5606946914816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126415235 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88792364 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384577869745682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394791335116800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚6699 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56780060 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126395197 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97318686 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97318688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32012304 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389183731839488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425011453713920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384577198268566 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88792807 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389183713836544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5606946914816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56780062 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012745 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5882093979136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389183713836544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425011429773824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5881926160896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844ü¤ÜÓ~—ßO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394790660751360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430893355934720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430893547693056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378990909418496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126415235 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚394791335116800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5586960327186 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5606946914816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126415235 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88792364 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384577869745682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394791335116800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚6699 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56780060 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_K†ggbatch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126395197 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97318686 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97318688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32012304 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389183731839488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425011453713920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384577198268566 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88792807 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389183713836544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5606946914816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56780062 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012745 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5882093979136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389183713836544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425011429773824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5881926160896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126433715 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223746353 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97332704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56788228 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425073080511488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430955870708736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32016836 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384633590311414 trainer/global_step‚6699 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430956056162304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126433715 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88805064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425073104451584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56788230 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394847833218048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884100974392517e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5607766342656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚5200.427091331 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚389240084878336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5882790197248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126413651 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚394848507596800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223766419 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389240066875392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126413651 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5607766342656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32017284 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5882951710720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 _step‚549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97332704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97332702 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚384632918834090 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379045813549056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844A5Úäð&šêM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126433715 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88805064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425073104451584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56788230 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394847833218048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884100974392517e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5607766342656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚5200.427091331 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚389240084878336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5882790197248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126413651 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚394848507596800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223766419 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389240066875392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126413651 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5607766342656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32017284 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5882951710720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 _step‚549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97332704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97332702 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚384632918834090 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379045813549056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126433715 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223746353 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97332704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56788228 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425073080511488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430955870708736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32016836 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384633590311414 trainer/global_step‚6699 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430956056162304¥‹À2›- _wandbruntime‚5200  _runtime‚5200…ä·YœÓ 1loss/FBt3/assignment_loss‚0.23596467385344122 2loss/SRqqt4/assignment_loss‚1.3309672622726514 1loss/FBt4/assignment_loss‚0.22519697961563775 /loss/FRt2/detection_loss‚2.4308241374638624 .loss/FBt4/detection_loss‚2.191164736236845 2loss/SRqqt2/assignment_loss‚1.3727733318305886 2loss/SRqqt3/assignment_loss‚0.9533258684859375 0loss/SRqqt3/detection_loss‚1.138311245681151 -loss/FBt1/detection_loss‚3.18172527551651 %loss/total_loss‚43.50971270582746 % _timestamp‚1.7884100974411647e+09 /loss/FRt1/assignment_loss‚3.347883320280484 0loss/SRqqt4/detection_loss‚1.712675214272279 .loss/FBt2/detection_loss‚4.325941309332848 epoch‚5 0loss/FRt2/assignment_loss‚3.0893069189421984 2loss/FBt2/assignment_loss‚0.027652718359604478 -loss/FRt1/detection_loss‚2.92941349159394 1loss/SRqqt2/detection_loss‚1.7663042212526003 /loss/FBt3/detection_loss‚0.6414936999020883 trainer/global_step‚6699 0loss/FRt3/assignment_loss‚1.8780084722920467 1loss/SRqqt1/detection_loss‚2.3680363874281607 _runtime‚5200.428554731 _step‚550 /loss/FRt4/assignment_loss‚2.390531600940795 .loss/FRt4/detection_loss‚2.328274047800473 2loss/SRqqt1/assignment_loss‚1.9592367509199726 1loss/FBt1/assignment_loss‚0.27962839824613184 /loss/FRt3/detection_loss‚1.4050726433072174èÑÇç^Ø 0loss/FRt3/assignment_loss‚1.8780084722920467 1loss/SRqqt1/detection_loss‚2.3680363874281607 _runtime‚5200.428554731 _step‚550 /loss/FRt4/assignment_loss‚2.390531600940795 .loss/FRt4/detection_loss‚2.328274047800473 2loss/SRqqt1/assignment_loss‚1.9592367509199726 1loss/FBt1/assignment_loss‚0.27962839824613184 /loss/FRt3/detection_loss‚1.4050726433072174 1loss/FBt3/assignment_loss‚0.23596467385344122 2loss/SRqqt4/assignment_loss‚1.3309672622726514 1loss/FBt4/assignment_loss‚0.22519697961563775 /loss/FRt2/detection_loss‚2.4308241374638624 .loss/FBt4/detection_loss‚2.191164736236845 2loss/SRqqt2/assignment_loss‚1.3727733318305886 2loss/SRqqt3/assignment_loss‚0.9533258684859375 0loss/SRqqt3/detection_loss‚1.138311245681151 -loss/FBt1/detection_loss‚3.18172527551651 %loss/total_loss‚43.50971270582746 % _timestamp‚1.7884100974411647e+09 /loss/FRt1/assignment_loss‚3.347883320280484 0loss/SRqqt4/detection_loss‚1.712675214272279 .loss/FBt2/detection_loss‚4.325941309332848 epoch‚5 0loss/FRt2/assignment_loss‚3.0893069189421984 2loss/FBt2/assignment_loss‚0.027652718359604478 -loss/FRt1/detection_loss‚2.92941349159394 1loss/SRqqt2/detection_loss‚1.7663042212526003 /loss/FBt3/detection_loss‚0.6414936999020883 trainer/global_step‚6699¦ð6D6ž‚(:… öñãÔȉ± gpu.0.gpu‚34.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.742 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"•ÂuŸ‚(:ê öñãÔ„±Ì ' disk.nvme0c0n1.out‚37609.9521484375 disk.rbd1.out‚ 24.9345703125# memory_percent‚94.0166711807251 proc.cpu.threads‚17 network.sent‚ 1.5699872e+07 network.recv‚ 1.928516e+06 disk.sda.in‚ 0.1953125 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 77.74609375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2802.8671875 disk.rbd0.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚37609.9521484375" disk.dm-0.out‚37609.9521484375 disk.dm-1.in‚ 0.1953125% disk./.usageGB‚246.25525665283203 disk.sda.out‚ 252.4765625 disk.sda3.out‚ 252.4765625 disk.drbd2080.out‚0 disk.drbd2265.in‚0) disk./.usagePercent‚16.52639689589431! disk.nvme0c0n1.in‚ 77.74609375 disk.drbd1716.in‚0* proc.memory.percent‚17.107343673706055 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 77.74609375 disk.dm-1.out‚ 252.4765625 cpu‚25.33862111129848 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 980.30859375¼™,%’ ‚(:‡ …òãÔ’ñ¦ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.856' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.1712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G' \w¡‚(:ì …òãÔ ³š % disk./.usageGB‚246.29132461547852 disk.drbd2080.in‚0 disk.sda3.out‚ 253.25390625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.dm-1.in‚ 0.1953125 disk.sda1.in‚0 disk.nvme0n1.in‚ 77.74609375 disk.dm-0.in‚ 77.74609375! disk.dm-0.out‚37697.333984375 disk.drbd2265.out‚0 network.sent‚ 1.5808751e+07 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 253.25390625 disk.rbd0.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚37697.333984375 disk.rbd1.out‚ 25.2548828125 disk.drbd2265.in‚0' proc.memory.availableMB‚ 980.2890625" proc.memory.rssMB‚ 2802.8671875 proc.cpu.threads‚17* disk./.usagePercent‚16.528817447130553! disk.nvme0c0n1.in‚ 77.74609375* proc.memory.percent‚17.107343673706055 disk.sda2.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚37697.333984375 disk.sda.in‚ 0.1953125 disk.sda.out‚ 253.25390625$ memory_percent‚94.01679039001465 network.recv‚ 1.936107e+06 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 0.1953125 cpu‚25.337799659674527²C·¦‘¢‚(:† ”òãÔ·“ž gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚125.445' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.089 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öî8,x£‚(:í ”òãÔÔ¶‚ % disk.nvme0n1.out‚37753.9853515625 network.recv‚ 1.939891e+06 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2802.8671875$ disk./.usageGB‚246.3812370300293! disk.nvme0c0n1.in‚ 77.74609375 disk.rbd1.in‚0 disk.dm-0.in‚ 77.74609375 network.sent‚ 1.5814025e+07 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 77.74609375( proc.memory.availableMB‚ 980.28515625 disk.dm-1.out‚ 253.98046875 disk.sda.in‚ 0.1953125 disk.drbd1716.in‚0 disk.rbd1.out‚ 25.2548828125 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 253.98046875 disk.sda3.out‚ 253.98046875 disk.rbd0.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚37753.9853515625" disk.dm-0.out‚37753.9853515625$ memory_percent‚94.01681423187256( disk./.usagePercent‚16.5348515447126 disk.drbd2266.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚17.107343673706055 cpu‚25.337008352959835 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.out‚0h6é2¤- _wandbruntime‚5236  _runtime‚5236Iá&'ɥà % _timestamp‚1.7884101330786757e+09 _runtime‚5236.066102651 _step‚551 ' lr-AdamW/pg1‚0.00013357913150388714 ' lr-AdamW/pg2‚0.00013357913150388714 trainer/global_step‚6749äÎ¦È ' lr-AdamW/pg1‚0.00013357913150388714 ' lr-AdamW/pg2‚0.00013357913150388714 trainer/global_step‚6749 % _timestamp‚1.7884101330786757e+09 _runtime‚5236.066102651 _step‚551§A£2§- _wandbruntime‚5236  _runtime‚5236> =¨ÚO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5627141228838 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_po_bGÍ%ol.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5647932720640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127321249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5925093824000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚381736115946496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚434018898954240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397649972865024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚5236.067356646 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89428158 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397649298499584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5924932297216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392001365778944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392001383781888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57188462 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434019084421120 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5648589083136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225360775 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98019486 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32239254 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225340735 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98019488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127341287 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89427714 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57188460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32239696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387363240135974 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397649972865024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428093990597120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428093966657024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392001383781888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127321249 $ _timestamp‚1.788410133079585e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397649298499584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381736098907136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98019488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5647932720640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5648589083136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98019486 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392001365778944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5647932720640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127321249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5925093824000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚381736115946496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚434018898954240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397649972865024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚5236.067356646 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89428158 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397649298499584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5924932297216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392001365778944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392001383781888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57188462 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434019084421120 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5648589083136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225360775 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98019486 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32239254 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225340735 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98019488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127341287 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89427714 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57188460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32239696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387363240135974 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397649972865024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428093990597120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428093966657024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392001383781888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127321249 $ _timestamp‚1.788410133079585e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397649298499584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381736098907136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98019488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5647932720640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5648589083136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98019486 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392001365778944 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89440378 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225373205 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98033504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5648752148480 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5627957663802 trainer/global_step‚6749 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57196628 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5649408524288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225393271 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381791003037696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392057736820736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392057736820736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5925954269184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5649408524288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98033502 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127339703 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434081595603968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392057718817792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428155641334784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98033504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387418960701498 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381791020077056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884101340024586e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392057718817792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5648752148480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127339703 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225373205 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚553 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32243750 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127359767 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32244181 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57196630 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89440378 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225373205 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98033504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5648752148480 J;De #Ö$âviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5627957663802 trainer/global_step‚6749 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57196628 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5649408524288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225393271 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381791003037696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392057736820736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392057736820736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5925954269184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5649408524288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98033502 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127339703 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434081595603968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392057718817792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428155641334784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98033504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387418960701498 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381791020077056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884101340024586e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392057718817792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5648752148480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127339703 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225373205 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚553 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32243750 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127359767 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32244181 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964©et0Ç2­- _wandbruntime‚5236  _runtime‚5236aݳY®Ó 1loss/SRqqt4/detection_loss‚1.6441251749793688 .loss/FBt1/detection_loss‚2.409471534192562 1loss/FBt3/assignment_loss‚0.07728755516802019 /loss/FRt4/detection_loss‚2.4284060843612836 1loss/SRqqt1/detection_loss‚2.3345004428516734 /loss/FRt3/assignment_loss‚1.274673451979955 /loss/FRt4/assignment_loss‚2.803829723078272 2loss/SRqqt3/assignment_loss‚0.8065807292407209 2loss/SRqqt4/assignment_loss‚1.5487162948896487 % _timestamp‚1.7884101340043986e+09 /loss/FRt3/detection_loss‚1.6350213034699361 1loss/SRqqt3/detection_loss‚0.9199630800973285 -loss/FBt2/detection_loss‚2.58036825873635 1loss/FBt2/assignment_loss‚0.13076897198334336 /loss/FRt1/assignment_loss‚3.509554874151945 /loss/FRt1/detection_loss‚2.9836655408143997 0loss/SRqqt2/detection_loss‚1.745103475045074 _step‚554 0loss/FBt1/assignment_loss‚0.3242717736575287 /loss/FRt2/detection_loss‚2.2887690962280933 %loss/total_loss‚43.31544975016102 epoch‚5 trainer/global_step‚6749 0loss/FRt2/assignment_loss‚3.2029945908523185 2loss/SRqqt1/assignment_loss‚1.8689179673339382 0loss/FBt4/assignment_loss‚0.8892982806385893 /loss/FBt3/detection_loss‚0.6277321397600805 .loss/FBt4/detection_loss‚3.575539310773214 _runtime‚5236.991733449 2loss/SRqqt2/assignment_loss‚1.7058900958773764Ño.^¯Ø 0loss/FRt2/assignment_loss‚3.2029945908523185 2loss/SRqqt1/assignment_loss‚1.8689179673339382 0loss/FBt4/assignment_loss‚0.8892982806385893 /loss/FBt3/detection_loss‚0.6277321397600805 .loss/FBt4/detection_loss‚3.575539310773214 _runtime‚5236.991733449 2loss/SRqqt2/assignment_loss‚1.7058900958773764 1loss/SRqqt4/detection_loss‚1.6441251749793688 .loss/FBt1/detection_loss‚2.409471534192562 1loss/FBt3/assignment_loss‚0.07728755516802019 /loss/FRt4/detection_loss‚2.4284060843612836 1loss/SRqqt1/detection_loss‚2.3345004428516734 /loss/FRt3/assignment_loss‚1.274673451979955 /loss/FRt4/assignment_loss‚2.803829723078272 2loss/SRqqt3/assignment_loss‚0.8065807292407209 2loss/SRqqt4/assignment_loss‚1.5487162948896487 % _timestamp‚1.7884101340043986e+09 /loss/FRt3/detection_loss‚1.6350213034699361 1loss/SRqqt3/detection_loss‚0.9199630800973285 -loss/FBt2/detection_loss‚2.58036825873635 1loss/FBt2/assignment_loss‚0.13076897198334336 /loss/FRt1/assignment_loss‚3.509554874151945 /loss/FRt1/detection_loss‚2.9836655408143997 0loss/SRqqt2/detection_loss‚1.745103475045074 _step‚554 0loss/FBt1/assignment_loss‚0.3242717736575287 /loss/FRt2/detection_loss‚2.2887690962280933 %loss/total_loss‚43.31544975016102 epoch‚5 trainer/global_step‚6749ª!ý¬°‚(:’ £òãÔáø— gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.438' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.087599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û3Y9u±‚(:ê £òãÔÑêÛ $ memory_percent‚94.01726722717285 cpu‚25.33615775025742 disk.drbd2266.out‚0* disk./.usagePercent‚16.542842051871304! disk.nvme0c0n1.in‚ 77.74609375& disk.nvme0c0n1.out‚37822.716796875 disk.drbd1716.in‚0 disk.sda.out‚ 254.62890625 disk.sda3.in‚ 0.1953125 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 254.62890625 disk.drbd2265.in‚0 disk.drbd2266.in‚0' proc.memory.availableMB‚ 980.2109375 network.recv‚ 1.944219e+06 disk.drbd2265.out‚0 disk.drbd2080.out‚0 network.sent‚ 1.5820048e+07 disk.drbd1716.out‚0 disk.rbd1.out‚ 25.287109375" proc.memory.rssMB‚ 2802.8671875 disk.sda3.out‚ 254.62890625 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0% disk./.usageGB‚246.50030136108398 disk.sda1.out‚0 disk.nvme0n1.in‚ 77.74609375 disk.dm-0.in‚ 77.74609375* proc.memory.percent‚17.107343673706055 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.in‚0! disk.dm-0.out‚37822.716796875 disk.drbd2080.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚37822.716796875 disk.sda.in‚ 0.1953125ØmÞ‘²‚(:† ²òãÔúó› gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.945' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.989 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êŒÍ^m³‚(:â ²òãÔ»Òü  disk.sda2.in‚0 network.recv‚ 1.951645e+06 disk.nvme0n1.in‚ 77.74609375 disk.dm-1.out‚ 255.15625 disk.drbd2265.out‚0 disk.sda.out‚ 255.15625 disk.rbd0.in‚0 disk.rbd1.out‚ 25.380859375* proc.memory.percent‚17.107343673706055 disk.rbd0.out‚0& disk.nvme0c0n1.out‚37876.681640625 cpu‚25.33533387970941 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0$ memory_percent‚94.01571750640869( proc.memory.availableMB‚ 980.46484375% disk./.usageGB‚246.54415893554688 disk.dm-0.in‚ 77.74609375 disk.sda3.in‚ 0.1953125 disk.drbd1716.out‚0 network.sent‚ 1.5928521e+07! disk.dm-0.out‚37876.681640625 disk.drbd2266.out‚0 disk.drbd2080.out‚0! disk.nvme0c0n1.in‚ 77.74609375 disk.sda1.in‚0* disk./.usagePercent‚16.545785370492467 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2802.8671875 disk.sda2.out‚0$ disk.nvme0n1.out‚37876.681640625 disk.sda3.out‚ 255.15625 disk.sda1.out‚0ê%N2´- _wandbruntime‚5272  _runtime‚5272€;qɵà _step‚555 ' lr-AdamW/pg1‚0.00013159503588167199 ' lr-AdamW/pg2‚0.00013159503588167199 trainer/global_step‚6799 % _timestamp‚1.7884101697305264e+09 _runtime‚5272.717991539ð™;åÎ¶È ' lr-AdamW/pg1‚0.00013159503588167199 ' lr-AdamW/pg2‚0.00013159503588167199 trainer/global_step‚6799 % _timestamp‚1.7884101697305264e+09 _runtime‚5272.717991539 _step‚555«žs¼2·- _wandbruntime‚5272  _runtime‚5272ªÿ|:á'¸ÛO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚400508610613248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394819035724288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884101697312462e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚394819017721344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5968087635968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400507936247808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128267339 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390149953480498 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98720286 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32466256 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384481305435136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5668631006002 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226987627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431176503540224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390149282003382 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394819035724288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5667976568246 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128267339 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90063560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57596860 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384481322474496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57596862 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128247301 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5968244968448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400508610613248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400508610613248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚400508610613248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394819035724288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884101697312462e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚394819017721344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5968087635968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400507936247808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128267339 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390149953480498 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98720286 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32466256 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384481305435136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5668631006002 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226987627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431176503540224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390149282003382 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394819035724288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5667976568246 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128267339 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90063560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57596860 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384481322474496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_p%Áÿool.allocated‚57596862 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128247301 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5968244968448 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5690394330112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690394330112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32470807 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394875370760192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5689737954304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227000057 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98734302 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚431238178217984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390205674046230 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227000057 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394875388763136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98734304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90075835 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5689737954304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227020123 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98734302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128285819 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390205002568906 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5669447441174 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚394875388763136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128265755 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437207104223744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400565783093248 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98734304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128285819 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5968949945856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788410170413508e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5969111459328 _runtime‚5273.401483563 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32471242 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394875370760192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128265755 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32471242 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394875370760192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128265755 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5690394330112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690394330112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32470807 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394875370760192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5689737954304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227000057 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98734302 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚431238178217984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390205674046230 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227000057 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394875388763136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98734304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90075835 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5689737954304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227020123 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98734302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128285819 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390205002568906 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5669447441174 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚394875388763136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128265755 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437207104223744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400565783093248 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98734304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128285819 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5968949945856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788410170413508e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5969111459328 _runtime‚5273.401483563 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560­7Múr2½- _wandbruntime‚5273  _runtime‚5273ú£5Ä`¾Ú 0loss/FRt1/assignment_loss‚3.5120282267269336 2loss/SRqqt1/assignment_loss‚2.2345865853130817 1loss/FBt1/assignment_loss‚0.05069729458244348 /loss/FRt1/detection_loss‚2.7124087590920296 .loss/FRt2/detection_loss‚2.339150032830751 1loss/SRqqt3/detection_loss‚0.9914432935479839 2loss/SRqqt4/assignment_loss‚1.3369098277214695 1loss/SRqqt2/detection_loss‚2.4550684287267575 %loss/total_loss‚42.70626531246232 trainer/global_step‚6799 0loss/FRt4/assignment_loss‚2.8007917948391126 /loss/FRt3/detection_loss‚1.6482459120452404 _step‚558 % _timestamp‚1.7884101704154882e+09 0loss/FRt2/assignment_loss‚2.9638782938321433 2loss/SRqqt2/assignment_loss‚1.5544518021976246 2loss/SRqqt3/assignment_loss‚1.2044964917635514 1loss/FBt2/assignment_loss‚0.22337458797264845 .loss/FRt4/detection_loss‚2.280325873390488 epoch‚5 0loss/FRt3/assignment_loss‚1.1142940698785986 /loss/FBt1/detection_loss‚2.7412179058248345 1loss/SRqqt4/detection_loss‚1.5652588687691034 0loss/FBt3/assignment_loss‚0.1704995316181413 2loss/FBt4/assignment_loss‚0.053510796797292474 1loss/SRqqt1/detection_loss‚2.2542637358109157 .loss/FBt2/detection_loss‚3.166340911388397 /loss/FBt3/detection_loss‚0.4995934931053357 /loss/FBt4/detection_loss‚2.8334287946874444 _runtime‚5273.402873766£yƒôe¿ß 0loss/FRt4/assignment_loss‚2.8007917948391126 /loss/FRt3/detection_loss‚1.6482459120452404 _step‚558 % _timestamp‚1.7884101704154882e+09 0loss/FRt2/assignment_loss‚2.9638782938321433 2loss/SRqqt2/assignment_loss‚1.5544518021976246 2loss/SRqqt3/assignment_loss‚1.2044964917635514 1loss/FBt2/assignment_loss‚0.22337458797264845 .loss/FRt4/detection_loss‚2.280325873390488 epoch‚5 0loss/FRt3/assignment_loss‚1.1142940698785986 /loss/FBt1/detection_loss‚2.7412179058248345 1loss/SRqqt4/detection_loss‚1.5652588687691034 0loss/FBt3/assignment_loss‚0.1704995316181413 2loss/FBt4/assignment_loss‚0.053510796797292474 1loss/SRqqt1/detection_loss‚2.2542637358109157 .loss/FBt2/detection_loss‚3.166340911388397 /loss/FBt3/detection_loss‚0.4995934931053357 /loss/FBt4/detection_loss‚2.8334287946874444 _runtime‚5273.402873766 0loss/FRt1/assignment_loss‚3.5120282267269336 2loss/SRqqt1/assignment_loss‚2.2345865853130817 1loss/FBt1/assignment_loss‚0.05069729458244348 /loss/FRt1/detection_loss‚2.7124087590920296 .loss/FRt2/detection_loss‚2.339150032830751 1loss/SRqqt3/detection_loss‚0.9914432935479839 2loss/SRqqt4/assignment_loss‚1.3369098277214695 1loss/SRqqt2/detection_loss‚2.4550684287267575 %loss/total_loss‚42.70626531246232 trainer/global_step‚6799®Â*@ ‘À‚(:† ÁòãÔ¥‘ï gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.517 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•/ zyÁ‚(:î ÁòãÔÁà—  disk.sda2.in‚0# memory_percent‚94.0159797668457 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125& proc.memory.availableMB‚ 980.421875" proc.memory.rssMB‚ 2802.8671875 network.recv‚ 1.956198e+06 network.sent‚ 1.5934833e+07 disk.dm-1.out‚ 255.63671875 disk.drbd2080.in‚0* proc.memory.percent‚17.107343673706055 disk.drbd2265.in‚0 disk.sda.out‚ 255.63671875( disk.nvme0c0n1.out‚37955.34423828125 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 77.74609375 cpu‚25.334534892212837$ disk./.usageGB‚246.5910758972168 disk.rbd1.out‚ 25.380859375& disk.nvme0n1.out‚37955.34423828125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 77.74609375 disk.dm-0.in‚ 77.74609375 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚16.548934007156085 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 255.63671875 disk.drbd2266.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚37955.34423828125 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0‹—f¹’‚(:‡ ÐòãÔ𿱠gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.417' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'Î ËoÂ(:ä ÐòãÔ©Ê€  cpu‚25.33373439392203 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.dm-0.in‚87.6875 disk.sda3.out‚ 256.26953125 network.recv‚ 1.96279e+06 network.sent‚ 1.6042862e+07& disk.nvme0n1.out‚38005.18798828125 disk.dm-1.out‚ 256.26953125 disk.sda.out‚ 256.26953125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚87.6875 disk.drbd2265.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚38005.18798828125$ memory_percent‚94.01590824127197( proc.memory.availableMB‚ 980.43359375* disk./.usagePercent‚16.550224029121125 disk.rbd1.out‚ 25.443359375 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125* proc.memory.percent‚17.107343673706055 disk.drbd1716.in‚0# disk.dm-0.out‚38005.18798828125 disk.rbd0.in‚0 disk.drbd2080.out‚0% disk./.usageGB‚246.61029815673828 disk.rbd1.in‚0 disk.nvme0n1.in‚87.6875 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2802.8671875ƒA¦2Ä- _wandbruntime‚5309  _runtime‚5309xÊõÉÅà ' lr-AdamW/pg2‚0.00012961217699807522 trainer/global_step‚6849 % _timestamp‚1.7884102061386478e+09 _runtime‚5309.126164468 _step‚559 ' lr-AdamW/pg1‚0.00012961217699807522¬YNEÎÆÈ trainer/global_step‚6849 % _timestamp‚1.7884102061386478e+09 _runtime‚5309.126164468 _step‚559 ' lr-AdamW/pg1‚0.00012961217699807522 ' lr-AdamW/pg2‚0.00012961217699807522¯÷6ì2Ç- _wandbruntime‚5309  _runtime‚5309l–±‰ÈÚO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99421088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32694754 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228614479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403367248361472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397636687666688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403367248361472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129193391 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90699562 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5729904332288 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.curren&•¤žWt‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403366573996032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397636669663744 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99421086 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129193391 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5730560694784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5729904332288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 % _timestamp‚1.7884102061393821e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58005262 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387226511963136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5709466345410 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434259040423424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129173353 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5730560694784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434259064363520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392935995347906 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403366573996032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90700016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58005260 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440270135604736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6011265096704 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228594439 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99421088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392935323870790 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403366573996032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397636669663744 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99421086 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129193391 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5730560694784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5729904332288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 % _timestamp‚1.7884102061393821e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58005262 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387226511963136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5709466345410 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434259040423424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129173353 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5730560694784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434259064363520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392935995347906 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403366573996032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90700016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58005260 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440270135604736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6011265096704 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228594439 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99421088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392935323870790 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99421088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32694754 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228614479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403367248361472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397636687666688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403367248361472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129193391 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90699562 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5729904332288 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397693040705536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6011974985216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99435104 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6012128110080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58013430 trainer/global_step‚6849 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397693022702592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434320715101184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397693022702592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403424420841472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440332666146304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99435104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440332843211264 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884102068234751e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32698744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434320691161088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129191807 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5730723760128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90712172 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5710282780582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128òÇË„ÌêM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434320691161088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129191807 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5730723760128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90712172 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5710282780582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99435102 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32699191 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397693040705536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6011974985216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99435104 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6012128110080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58013430 trainer/global_step‚6849 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397693022702592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.curre‹Y[òl nt‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434320715101184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397693022702592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403424420841472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440332666146304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99435104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440332843211264 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884102068234751e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32698744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0±Ž«ˆ2Í- _wandbruntime‚5309  _runtime‚53098a_ÎÙ 2loss/SRqqt3/assignment_loss‚0.7414254795219148 1loss/FBt1/assignment_loss‚0.32417343927081677 1loss/FBt2/assignment_loss‚0.24554695809880775 /loss/FRt1/detection_loss‚2.5637419039294835 1loss/SRqqt4/detection_loss‚1.7886007726192474 _step‚562 % _timestamp‚1.7884102068253286e+09 .loss/FRt2/detection_loss‚2.104899601733566 1loss/SRqqt1/detection_loss‚2.1736300684310295 1loss/SRqqt3/detection_loss‚1.0514128133654594 %loss/total_loss‚40.91842689899203 0loss/FRt1/assignment_loss‚3.0650247051602317 0loss/FRt4/assignment_loss‚2.4274512898921965 1loss/SRqqt1/assignment_loss‚1.945023402168944 2loss/SRqqt2/assignment_loss‚1.5259102149629744 1loss/FBt3/assignment_loss‚0.16143474491935023 /loss/FRt4/detection_loss‚2.5140110272169114 trainer/global_step‚6849 0loss/FRt3/assignment_loss‚0.8839252548558372 2loss/SRqqt4/assignment_loss‚1.2618515722605648 1loss/FBt4/assignment_loss‚0.36984216533819564 1loss/SRqqt2/detection_loss‚1.7698726373708855 /loss/FBt2/detection_loss‚3.1550782124201455 0loss/FRt2/assignment_loss‚2.5351588992362326 .loss/FRt3/detection_loss‚1.837875353904175 /loss/FBt4/detection_loss‚2.3420079708099366 .loss/FBt1/detection_loss‚3.589769017696381 /loss/FBt3/detection_loss‚0.5407593938087424 epoch‚5 _runtime‚5309.812714326ö“zdÏÞ 2loss/SRqqt4/assignment_loss‚1.2618515722605648 1loss/FBt4/assignment_loss‚0.36984216533819564 1loss/SRqqt2/detection_loss‚1.7698726373708855 /loss/FBt2/detection_loss‚3.1550782124201455 0loss/FRt2/assignment_loss‚2.5351588992362326 .loss/FRt3/detection_loss‚1.837875353904175 /loss/FBt4/detection_loss‚2.3420079708099366 .loss/FBt1/detection_loss‚3.589769017696381 /loss/FBt3/detection_loss‚0.5407593938087424 epoch‚5 _runtime‚5309.812714326 2loss/SRqqt3/assignment_loss‚0.7414254795219148 1loss/FBt1/assignment_loss‚0.32417343927081677 1loss/FBt2/assignment_loss‚0.24554695809880775 /loss/FRt1/detection_loss‚2.5637419039294835 1loss/SRqqt4/detection_loss‚1.7886007726192474 _step‚562 % _timestamp‚1.7884102068253286e+09 .loss/FRt2/detection_loss‚2.104899601733566 1loss/SRqqt1/detection_loss‚2.1736300684310295 1loss/SRqqt3/detection_loss‚1.0514128133654594 %loss/total_loss‚40.91842689899203 0loss/FRt1/assignment_loss‚3.0650247051602317 0loss/FRt4/assignment_loss‚2.4274512898921965 1loss/SRqqt1/assignment_loss‚1.945023402168944 2loss/SRqqt2/assignment_loss‚1.5259102149629744 1loss/FBt3/assignment_loss‚0.16143474491935023 /loss/FRt4/detection_loss‚2.5140110272169114 trainer/global_step‚6849 0loss/FRt3/assignment_loss‚0.8839252548558372²‘ï[bœÐ‚(:‘ ßòãÔê¦ö  gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.146' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚36.22919999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÝÁhÑ‚(:Ý ßòãÔÃ¥ƒ  disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚87.6875 disk.sda3.out‚ 256.75390625* proc.memory.percent‚17.107343673706055 proc.cpu.threads‚17 disk.drbd1716.out‚0 network.recv‚ 1.967323e+06 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚87.6875$ disk.nvme0n1.out‚38083.146484375# memory_percent‚94.0159559249878 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda3.in‚ 0.1953125( proc.memory.availableMB‚ 980.42578125 cpu‚25.332879798781843 network.sent‚ 1.6048987e+07 disk.sda2.out‚0& disk.nvme0c0n1.out‚38083.146484375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd2265.out‚0) disk./.usagePercent‚16.55322085331694 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.1953125 disk.sda.out‚ 256.75390625! disk.dm-0.out‚38083.146484375 disk.nvme0c0n1.in‚87.6875 disk.drbd1716.in‚0 disk.rbd1.out‚ 25.443359375$ disk./.usageGB‚246.6549530029297 disk.drbd2266.in‚0 disk.dm-1.out‚ 256.75390625" proc.memory.rssMB‚ 2802.8671875 disk.rbd0.out‚0ך„ÞÒ‚(:’ îòãÔÐÆŒ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.349' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.069799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆiò>Ó‚(:ö îòãÔÒÏÕ  disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda.out‚ 257.38671875 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2802.8671875 disk.dm-0.in‚ 1144.24609375 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1144.24609375# disk.nvme0c0n1.in‚ 1144.24609375 disk.sda3.out‚ 257.38671875 disk.sda1.in‚0 network.recv‚ 1.974388e+06& disk.nvme0n1.out‚38227.57080078125# disk.dm-0.out‚38227.57080078125 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚38227.57080078125 disk.dm-1.in‚ 0.1953125 disk.drbd2080.in‚0' proc.memory.availableMB‚ 1005.421875* proc.memory.percent‚17.107343673706055) disk./.usagePercent‚16.55974494931952 disk.rbd1.in‚0 disk.rbd1.out‚ 25.6552734375 disk.dm-1.out‚ 257.38671875# memory_percent‚93.8633918762207 cpu‚25.332099957090165 network.sent‚ 1.6157424e+07% disk./.usageGB‚246.75216674804688 disk.drbd2265.in‚0 disk.drbd2266.out‚0`ÜÖ—’Ô‚(:‡ ýòãÔÌð gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EøuÕ‚(:ê ýòãÔÕ‹‡  disk.drbd1716.in‚0 disk.dm-1.out‚ 259.86328125 disk.sda3.out‚ 259.86328125 disk.sda1.in‚0& disk.nvme0n1.out‚38317.14111328125) proc.memory.availableMB‚ 1005.41796875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 25.6552734375 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 259.86328125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 1166.4375# disk.dm-0.out‚38317.14111328125" proc.memory.rssMB‚ 2802.8671875* disk./.usagePercent‚16.564722503807754 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0* proc.memory.percent‚17.107343673706055 network.sent‚ 1.616376e+07 disk.sda2.in‚0 disk.drbd2265.out‚0$ memory_percent‚93.86341571807861 cpu‚25.3312706770171% disk./.usageGB‚246.82633590698242 disk.rbd0.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚38317.14111328125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1166.4375 disk.dm-0.in‚ 1166.4375 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 network.recv‚ 1.97907e+06 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0æuW2Ö- _wandbruntime‚5345  _runtime‚5345ˆoblÈ× ' lr-AdamW/pg1‚0.00012763096807791622 ' lr-AdamW/pg2‚0.00012763096807791622 trainer/global_step‚6899 $ _timestamp‚1.788410242675554e+09 _runtime‚5345.662985635 _step‚563&ŒÚÍØÇ ' lr-AdamW/pg1‚0.00012763096807791622 ' lr-AdamW/pg2‚0.00012763096807791622 trainer/global_step‚6899 $ _timestamp‚1.788410242675554e+09 _runtime‚5345.662985635 _step‚563³Ùòì2Ù- _wandbruntime‚5345  _runtime‚5345`Ö‚>ä'ÚÞO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400454321606144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443396058788352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130119443 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130119443 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406225211744256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389971718491136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100121888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230241331 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443395864932864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58413660 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054457541632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395722037215314 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389971735530496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5750301684818 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395721365738198 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770890138112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788410242676333e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406225886109696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6054287626240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100121886 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91335382 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5749647247062 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400454339609088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5771546500608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437341577306624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230221291 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100121888o&xé'ÛãO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770890138112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788410242676333e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406225886109696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6054287626240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100121886 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91335382 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5749647247062 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400454339609088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5771546500608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437341577306624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230221291 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100121888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400454321606144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443396058788352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130119443 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130119443 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406225211744256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389971718491136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100121888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230241331 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443395864932864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58413660 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054457541632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395722037215314 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389971735530496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5750301684818 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395721365738198´ï\ù2Ü- _wandbruntime‚5346  _runtime‚5346  2ÝèM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6055150637568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 $ _timestamp‚1.788410243361336e+09 _step‚565 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230253761 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚443458564135424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395777086303722 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5750463682026 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130137923 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130117859 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100135904 VHDeviceStatsMonitor.on_train_batch_end1òu¥î /allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130137923 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58421830 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390026639661056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395777757781046 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5771709565952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230253761 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58421828 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437403228044288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚5346.349893523 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5772365941760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5751118119990 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5771709565952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100135904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91347606 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406282384210944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400510692647936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400510674644992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406283058589696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32926235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390026622621696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443458378681856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230273827 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6055312151040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100135902 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437403251984384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 trainer/global_step‚6899ïF?ïó&ÞíM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400510692647936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5772365941760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100135902 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91348065 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400510674644992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32925778 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230273827 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6055312151040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100135902 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437403251984384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 trainer/global_step‚6899 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6055150637568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 $ _timestamp‚1.788410243361336e+09 _step‚565 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230253761 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚443458564135424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395777086303722 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5750463682026 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130137923 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130117859 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100135904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130137923 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58421830 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390026639661056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395777757781046 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5771709565952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230253761 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58421828 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437403228044288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚5346.349893523 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5772365941760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5751118119990 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5771709565952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100135904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91347606 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406282384210944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400510692647936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400510674644992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406283058589696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32926235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390026622621696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443458378681856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403271973548544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚403271991551488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6097676297728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231848143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131025457 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5790482586470 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33149495 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚5382.287112526 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131045495 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409084523857920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 trainer/global_step‚6949 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5811875943936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392716942058496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884102792994435e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446521628960768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409083849492480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440424114189824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5812532306432 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446521814427648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6097514770944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131025457 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231868183 J 22686 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91971555 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚409083849492480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33149945 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398508079082722 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446521628960768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409083849492480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440424114189824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5812532306432 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446521814427648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6097514770944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131025457 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231868183 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403271973548544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚403271991551488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6097676297728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231848143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131025457 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5790482586470 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33149495 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚5382.287112526 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131045495 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409084523857920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 trainer/global_step‚6949¸¨Sv¥2ì- _wandbruntime‚5382  _runtime‚5382û ±ï&íéM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91984727 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚6949 _runtime‚5382.969034803 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100836704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131063975 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91984270 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231900679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33154042 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446584128230400 % _timestamp‚1.7884102799811285e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5812695371776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6098363302912 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392771829149696 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100836704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5812695371776 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392771846189056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398563799648454 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440485764927488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446584322072576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6098533204992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398563128171130 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231880613 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58830230 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403328326587392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5813351747584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131043911 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409141696337920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100836702 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚403328344590336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58830228 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33154497 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403328326587392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0êô&îîM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440485764927488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446584322072576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6098533204992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398563128171130 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231880613 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58830230 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403328326587392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5813351747584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131043911 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409141696337920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100836702 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚403328344590336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58830228 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33154497 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403328326587392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91984727 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚6949 _runtime‚5382.969034803 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100836704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131063975 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91984270 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231900679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33154042 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446584128230400 % _timestamp‚1.7884102799811285e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5812695371776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6098363302912 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392771829149696 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100836704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5812695371776 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392771846189056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398563799648454¹Ü$ïï2ï- _wandbruntime‚5382  _runtime‚5382"ç-`ðÐ 2loss/SRqqt1/assignment_loss‚1.9711658327529828 2loss/SRqqt3/assignment_loss‚0.96àev~ö17615038942959 0loss/SRqqt4/detection_loss‚1.675430138905843 _runtime‚5382.970135239 0loss/FRt4/assignment_loss‚2.7084008794406365 /loss/FRt1/detection_loss‚3.0075493935615785 /loss/FRt2/detection_loss‚2.0682410154105844 2loss/SRqqt4/assignment_loss‚1.6881139916367829 /loss/FBt1/assignment_loss‚0.698766564950347 /loss/FRt3/detection_loss‚1.3549531059106812 epoch‚5 0loss/FBt2/assignment_loss‚0.4769919440150261 0loss/FBt4/assignment_loss‚0.2680450927333108 1loss/SRqqt1/detection_loss‚2.5825703574551477 /loss/FBt3/detection_loss‚0.7157124261022545 1loss/FBt3/assignment_loss‚0.12187934777737715 0loss/SRqqt3/detection_loss‚0.906676244818502 .loss/FBt2/detection_loss‚2.989283573627472 %loss/total_loss‚44.94138187868926 /loss/FRt2/assignment_loss‚2.857606750064426 .loss/FBt1/detection_loss‚5.510477066040039 % _timestamp‚1.7884102799827788e+09 0loss/FRt1/assignment_loss‚3.9175162815278575 /loss/FRt3/assignment_loss‚1.104472509585321 1loss/SRqqt2/assignment_loss‚1.205794345587492 .loss/FRt4/detection_loss‚2.218799306400891 1loss/SRqqt2/detection_loss‚1.9513891302809423 .loss/FBt4/detection_loss‚1.979785076209477 trainer/global_step‚6949 _step‚570ö®?[ñÕ %loss/total_loss‚44.94138187868926 /loss/FRt2/assignment_loss‚2.857606750064426 .loss/FBt1/detection_loss‚5.510477066040039 % _timestamp‚1.7884102799827788e+09 0loss/FRt1/assignment_loss‚3.9175162815278575 /loss/FRt3/assignment_loss‚1.104472509585321 1loss/SRqqt2/assignment_loss‚1.205794345587492 .loss/FRt4/detection_loss‚2.218799306400891 1loss/SRqqt2/detection_loss‚1.9513891302809423 .loss/FBt4/detection_loss‚1.979785076209477 trainer/global_step‚6949 _step‚570 2loss/SRqqt1/assignment_loss‚1.9711658327529828 2loss/SRqqt3/assignment_loss‚0.9617615038942959 0loss/SRqqt4/detection_loss‚1.675430138905843 _runtime‚5382.970135239 0loss/FRt4/assignment_loss‚2.7084008794406365 /loss/FRt1/detection_loss‚3.0075493935615785 /loss/FRt2/detection_loss‚2.0682410154105844 2loss/SRqqt4/assignment_loss‚1.6881139916367829 /loss/FBt1/assignment_loss‚0.698766564950347 /loss/FRt3/detection_loss‚1.3549531059106812 epoch‚5 0loss/FBt2/assignment_loss‚0.4769919440150261 0loss/FBt4/assignment_loss‚0.2680450927333108 1loss/SRqqt1/detection_loss‚2.5825703574551477 /loss/FBt3/detection_loss‚0.7157124261022545 1loss/FBt3/assignment_loss‚0.12187934777737715 0loss/SRqqt3/detection_loss‚0.906676244818502 .loss/FBt2/detection_loss‚2.989283573627472ºøúæ[’ò‚(:‡ ªóãÔàÿŸ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þyvó‚(:ë ªóãÔÜ¦Þ % disk./.usageGB‚246.95417022705078 disk.rbd1.in‚0# disk.dm-0.out‚38547.51806640625 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 261.453125( disk.nvme0c0n1.out‚38547.51806640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚25.84375& disk.nvme0n1.out‚38547.51806640625 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 cpu‚25.328961043003936 network.sent‚ 1.6282884e+07 network.recv‚ 1.992224e+06 disk.drbd2265.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 1166.4765625 disk.nvme0n1.in‚ 1166.4765625 disk.rbd0.out‚0" proc.memory.rssMB‚ 2802.8671875 disk.dm-1.out‚ 261.453125 disk.sda2.out‚0$ memory_percent‚93.86613368988037 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0) disk./.usagePercent‚16.57330157228768 disk.dm-0.in‚ 1166.4765625 disk.sda.out‚ 261.453125 disk.drbd2266.out‚0) proc.memory.availableMB‚ 1004.97265625 disk.rbd0.in‚0* proc.memory.percent‚17.107343673706055‚¡1„ô‚(:’ ¹óãÔ³½Ù gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.024' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.004799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª·Öeqõ‚(:æ ¹óãÔÔ¦Ë  disk.nvme0n1.in‚ 1166.4765625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0$ disk./.usageGB‚246.9761505126953 disk.rbd1.out‚25.84375 disk.sda.out‚ 262.0859375 disk.drbd1716.in‚0% disk.nvme0n1.out‚38629.0439453125 disk.dm-0.in‚ 1166.4765625 disk.sda.in‚ 0.1953125 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 1.998517e+06 disk.dm-1.out‚ 262.0859375 disk.sda3.in‚ 0.1953125) disk./.usagePercent‚16.57477668769997" disk.dm-0.out‚38629.0439453125 disk.rbd0.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚38629.0439453125* proc.memory.percent‚17.107343673706055 disk.sda1.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0$ memory_percent‚93.86615753173828" proc.memory.rssMB‚ 2802.8671875 network.sent‚ 1.6391741e+07 disk.rbd1.in‚0 cpu‚25.32819097649867 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 262.0859375" disk.nvme0c0n1.in‚ 1166.4765625& proc.memory.availableMB‚ 1004.96875ëJÆ’ö‚(:‡ ÈóãÔ½•½ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³0öËo÷‚(:ä ÈóãÔ†¥Ö  disk.drbd2265.out‚0 disk.sda.out‚262.5625 disk.sda3.out‚262.5625 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 cpu‚25.327386894523638 network.recv‚ 2.002591e+06 disk.drbd2266.out‚0# disk.dm-0.out‚38688.51904296875& disk.nvme0n1.out‚38688.51904296875 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚262.5625* proc.memory.percent‚17.107343673706055 network.sent‚ 1.6397161e+07 disk.sda.in‚ 0.1953125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1166.4765625" proc.memory.rssMB‚ 2802.8671875% disk./.usageGB‚247.01461791992188 disk.rbd0.out‚0$ memory_percent‚93.86463165283203& proc.memory.availableMB‚ 1005.21875) disk./.usagePercent‚16.57735826767525" disk.nvme0c0n1.in‚ 1166.4765625( disk.nvme0c0n1.out‚38688.51904296875 disk.rbd1.out‚ 25.9921875 disk.dm-0.in‚ 1166.4765625 disk.dm-1.in‚ 0.1953125i†Ñ2ø- _wandbruntime‚5418  _runtime‚5418â/6ÆùÀ % _timestamp‚1.7884103157328668e+09 _runtime‚ 5418.72032389 _step‚571 & lr-AdamW/pg1‚0.0001236751512218955 & lr-AdamW/pg2‚0.0001236751512218955 trainer/global_step‚6999r~7wËúÅ & lr-AdamW/pg1‚0.0001236751512218955 & lr-AdamW/pg2‚0.0001236751512218955 trainer/global_step‚6999 % _timestamp‚1.7884103157328668e+09 _runtime‚ 5418.72032389 _step‚571»j›2û- _wandbruntime‚5418  _runtime‚5418Ò €å'üßO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6140842801152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5853518112256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33377739 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92607742 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406089625490944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449647493874176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443506651073024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101523486 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33377282 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚411943161606144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401294120950130 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131971547 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5852861749760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401293449473014 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411942487240704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131951509 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411943161606144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6141004327936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6999 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131951509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831972363634 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101523488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5852861749760 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853518112256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411942487240704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449647679341056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395462148586496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406089643493888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _runtime‚5418.721240298 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101523488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395462131547136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406089625490944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92608201 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0PÿLê'ýäO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406089643493888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _runtime‚5418.721240298 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101523488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395462131547136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406089625490944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92608201 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6140842801152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5853518112256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33377739 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92607742 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406089625490944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449647493874176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443506651073024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101523486 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33377282 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚411943161606144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401294120950130 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131971547 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5852861749760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401293449473014 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411942487240704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131951509 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411943161606144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6141004327936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6999 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131951509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831972363634 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101523488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5852861749760 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853518112256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411942487240704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449647679341056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395462148586496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928¼&ŒÔV2þ- _wandbruntime‚5419  _runtime‚5419¾Íæ2ÿçM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233507465 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5854337553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59238630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101537504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5854337553408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5419.421655006 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131969963 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131990027 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59238628 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33381883 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92620974 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚6999 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401349170038538 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131969963 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92620511 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449710014797312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395517035677696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚411999659707392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6141712986624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832788798806 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412000334086144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101537502 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414}Û§ò&€ìM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233507465 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5832134360842 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59238628 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33381883 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92620974 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚6999 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401349170038538 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131969963 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92620511 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449710014797312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395517035677696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚411999659707392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6141712986624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832788798806 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412000334086144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101537502 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233507465 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5854337553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59238630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101537504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5854337553408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5419.421655006 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131969963 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131990027 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93243024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6184039768064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132897599 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414801124988928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚5455.299831523 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408907277433344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132877561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102224286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59638862 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5872153265286 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 142652416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414801799354368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5894503918080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚404079491340422 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5872807703042 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788410352312178e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102224288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6184182420480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398207338075136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414801124988928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446589187956224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚7049 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚404080162817538 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5893847555584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33604599 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93243461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408907295436288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414801799354368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132897599 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102224286 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0l‚ãO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 142652416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_tÆŒ½Ìè%rain_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414801799354368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5894503918080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚404079491340422 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5872807703042 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788410352312178e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102224288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6184182420480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398207338075136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414801124988928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446589187956224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚7049 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚404080162817538 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5893847555584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33604599 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93243461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408907295436288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414801799354368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132897599 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102224286 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235121887 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚398207355114496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93243024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6184039768064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132897599 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414801124988928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚5455.299831523 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408907277433344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132877561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102224286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59638862 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5872153265286ÀtÉ­º2Ž- _wandbruntime‚5455  _runtime‚5455²xvÚî&èM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132896015 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446650838693888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5894666983424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414858297455616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33608596 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59647028 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚404135211905946 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408963630472192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5895323359232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚577 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414858971834368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446650862633984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132916079 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93255624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5872969700250 _runtime‚5455.993670816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408963648475136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235154383 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132896015 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93256071 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59647030 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6184887536128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7049 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102238302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33609041 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102238304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5873624138214 $ _timestamp‚1.788410353005871e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235134317 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2ûG= ó&íM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235154383 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132896015 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93256071 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59647030 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6184887536128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7049 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102238302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33609041 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102238304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5873624138214 $ _timestamp‚1.788410353005871e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235134317 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132896015 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446650838693888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5894666983424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414858297455616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33608596 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59647028 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚404135211905946 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408963630472192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5895323359232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚577 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414858971834368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446650862633984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132916079 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93255624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5872969700250 _runtime‚5455.993670816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408963648475136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Á칟™2‘- _wandbruntime‚5455  _runtime‚54557A®T’Î 2loss/SRqqt2/assignment_loss‚1.2680897983338904 .loss/FRt4/detection_loss‚2.458414190489313 /loss/FBt2/detection_loss‚2.4770784113142224 -loss/FBt4/detection_loss‚2.30797187771116 0loss/FRt3/assignment_loss‚1.2808916422785546 2loss/SRqqt1/assignment_loss‚1.7434496006095095 1loss/FBt1/assignment_loss‚0.01762592194912334 1loss/FBt2/assignment_loss‚0.07369789320768581 1loss/SRqqt3/detection_loss‚0.7978684057792028 .loss/FBt1/detection_loss‚4.392813662687938 2loss/SRqqt3/assignment_loss‚0.9619894522676865 0loss/FBt4/assignment_loss‚0.6563826456799039 /loss/FRt2/detection_loss‚1.9375091548031536 _runtime‚5455.995146419 /loss/FBt3/assignment_loss‚0.172089874510601 /loss/FRt3/detection_loss‚1.7741294358761943 -loss/FRt1/assignment_loss‚3.7232302069664 1loss/SRqqt4/detection_loss‚2.0944396007968034 /loss/FBt3/detection_loss‚0.6288871198728982 epoch‚5 _step‚578 $ _timestamp‚1.788410353007768e+09 /loss/FRt4/assignment_loss‚3.106090519739234 1loss/SRqqt1/detection_loss‚2.2284931132667944 2loss/SRqqt4/assignment_loss‚1.3706062014984048 /loss/FRt1/detection_loss‚2.9973899483680726 %loss/total_loss‚43.37701854732522 /loss/SRqqt2/detection_loss‚1.85393581904617 trainer/global_step‚7049 /loss/FRt2/assignment_loss‚3.053944050272306EÏõY“Ó 1loss/FBt1/assignment_loss‚0.01762592194912334 1loss/FBt2/assignment_loss‚0.07369789320768581 1loss/SRqqt3/detection_loss‚0.7978684057792028 .loss/FBt1/detection_loss‚4.392813662687938 2loss/SRqqt3/assignment_loss‚0.9619894522676865 0loss/FBt4/assignment_loss‚0.6563826456799039 /loss/FRt2/detection_loss‚1.9375091548031536 _runtime‚5455.995146419 /loss/FBt3/assignment_loss‚0.172089874510601 /loss/FRt3/detection_loss‚1.7741294358761943 -loss/FRt1/assignment_loss‚3.7232302069664 1loss/SRqqt4/detection_loss‚2.0944396007968034 /loss/FBt3/detection_loss‚0.6288871198728982 epoch‚5 _step‚578 $ _timestamp‚1.788410353007768e+09 /loss/FRt4/assignment_loss‚3.106090519739234 1loss/SRqqt1/detection_loss‚2.2284931132667944 2loss/SRqqt4/assignment_loss‚1.3706062014984048 /loss/FRt1/detection_loss‚2.9973899483680726 %loss/total_loss‚43.37701854732522 /loss/SRqqt2/detection_loss‚1.85393581904617 trainer/global_step‚7049 /loss/FRt2/assignment_loss‚3.053944050272306 2loss/SRqqt2/assignment_loss‚1.2680897983338904 .loss/FRt4/detection_loss‚2.458414190489313 /loss/FBt2/detection_loss‚2.4770784113142224 -loss/FBt4/detection_loss‚2.30797187771116 0loss/FRt3/assignment_loss‚1.2808916422785546 2loss/SRqqt1/assignment_loss‚1.7434496006095095Â>ØØ,(:‡ õóãÔÄæ¦ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.971' gpu.0.enforcedPowerLimitWattsSߤ‚500.0 gpu.0.powerPercent‚35.1942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª1Îy•‚(:î õóãÔÔŸ¼ & disk.nvme0n1.out‚38908.64697265625 disk.drbd2265.in‚0 disk.sda.out‚ 264.2265625 disk.sda1.in‚0( disk.nvme0c0n1.out‚38908.64697265625* proc.memory.percent‚17.107343673706055 network.recv‚ 2.017667e+06 disk.nvme0n1.in‚ 1166.4765625 disk.dm-0.in‚ 1166.4765625 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 26.1181640625 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 264.2265625 proc.cpu.threads‚17 network.sent‚ 1.6518081e+07$ disk./.usageGB‚247.0961570739746 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚16.58283042871964 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚93.86653900146484" proc.memory.rssMB‚ 2802.8671875 cpu‚25.32506754603173 disk.rbd1.in‚0 disk.dm-1.out‚ 264.2265625 disk.drbd2266.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 1166.4765625# disk.dm-0.out‚38908.64697265625 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0& proc.memory.availableMB‚ 1004.90625 disk.drbd1716.in‚0÷.;6’–‚(:‡ „ôãÔöæÉ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚131.978' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0níÝn—‚(:ã „ôãÔ“ªå  disk.rbd1.out‚ 26.1806640625 disk.rbd0.out‚0" proc.memory.rssMB‚ 2802.8671875* disk./.usagePercent‚16.584869272722326 disk.drbd2266.in‚0 disk.sda1.out‚0 network.sent‚ 1.6625955e+07 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚1166.75# disk.dm-0.out‚38970.64697265625 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0$ memory_percent‚93.86672973632812* proc.memory.percent‚17.107343673706055 disk.drbd1716.in‚0 network.recv‚ 2.023197e+06% disk./.usageGB‚247.12653732299805 disk.drbd2265.in‚0 disk.sda.out‚ 264.86328125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚1166.75 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 264.86328125 disk.drbd2266.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚38970.64697265625& disk.nvme0n1.out‚38970.64697265625 disk.dm-1.out‚ 264.86328125 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 disk.nvme0n1.in‚1166.75$ proc.memory.availableMB‚1004.875 cpu‚25.324314279359022 disk.drbd2265.out‚0(¶ÏJ’˜‚(:‡ “ôãÔ‰•› gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.512' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò»ß¾n™‚(:ã “ôãÔ¡¯í % disk.nvme0n1.out‚39043.4814453125 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚265.3125 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.6631913e+07 network.recv‚ 2.027437e+06 disk.sda3.in‚ 0.1953125" disk.dm-0.out‚39043.4814453125 cpu‚25.323558175621386 disk.sda.out‚265.3125 disk.sda3.out‚265.3125 disk.sda1.in‚0 disk.dm-0.in‚ 1167.703125 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0! disk.nvme0c0n1.in‚ 1167.703125) proc.memory.availableMB‚ 1004.87109375" proc.memory.rssMB‚ 2802.8671875 proc.cpu.threads‚17% disk./.usageGB‚247.15025329589844 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚39043.4814453125 disk.rbd1.out‚ 26.1806640625* proc.memory.percent‚17.107343673706055( disk./.usagePercent‚16.5864608715627 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0$ memory_percent‚93.86675357818604 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 1167.703125Îß 2š- _wandbruntime‚5491  _runtime‚5491«h_ɛà _step‚579 ' lr-AdamW/pg1‚0.00011973088268702373 ' lr-AdamW/pg2‚0.00011973088268702373 trainer/global_step‚7099 % _timestamp‚1.7884103887560837e+09 _runtime‚5491.743517896#]É…ÎœÈ ' lr-AdamW/pg2‚0.00011973088268702373 trainer/global_step‚7099 % _timestamp‚1.7884103887560837e+09 _runtime‚5491.743517896 _step‚579 ' lr-AdamW/pg1‚0.00011973088268702373Ãüµº2- _wandbruntime‚5491  _runtime‚5491³óíå'žßO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417659762737152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102925086 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93879598 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400952544603136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5935489723904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚449671724839424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455899012852736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚449671748779520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60047260 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912988604694 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60047262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102925086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5935489723904 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406865533207830 _runtime‚5491.744538772 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚580 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133803613 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102925088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411724929375744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33832777 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6227100449280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5913643042450 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400952561642496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6227264073216 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33832338 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884103887568464e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102925088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455898825288704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93880039 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411724947378688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133803613 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406866204684946 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411724947378688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5934833361408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133823651 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133823651 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417659762737152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417659762737152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102925086 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93879598 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400952544603136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5935489723904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚449671724839424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455899012852736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚449671748779520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60047260 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912988604694 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60047262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102925086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5935489723904 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406865533207830 _runtime‚5491.744538772 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚580 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133803613 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102925088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411724929375744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33832777 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6227100449280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5913643042450 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400952561642496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6227264073216 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33832338 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884103887568464e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102925088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455898825288704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93880039 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411724947378688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133803613 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406866204684946 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411724947378688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5934833361408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133823651 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133823651 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417659762737152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449733375577088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401007465773056 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5935652789248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411781282414592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5913805039658 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236761169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚449733399517184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133842131 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417717609582592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚406921253773354 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5936309165056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102939102 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93892817 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33836939 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93892367 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60055428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚455961526165504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚455961342809088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segme.ò9Ój nt.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102939104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914459477622 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5935652789248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411781300417536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401007448733696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411781300417536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚7099 % _timestamp‚1.7884103894388077e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33837387 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5492.426740427 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417716935203840 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417717609582592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5936309165056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133822067 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93892367 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60055428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚455961526165504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚455961342809088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102939104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914459477622 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5935652789248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411781300417536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401007448733696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411781300417536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚7099 % _timestamp‚1.7884103894388077e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33837387 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5492.426740427 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417716935203840 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417717609582592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5936309165056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133822067 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449733375577088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401007465773056 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5935652789248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411781282414592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5913805039658 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236761169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚449733399517184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133842131 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417717609582592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚406921253773354 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5936309165056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102939102 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93892817 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33836939 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270159048704 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103625888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420519074850816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5975819167232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134729665 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34061157 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5975819167232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34060716 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚459024612529664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788410425314927e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5976475529728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94516819 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6270326866944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5528.302610457 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134729665 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60455662 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5953823944102 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414542599321088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103625888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134749703 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452754261722624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420518400485376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103625886 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Õ¸f ¯ãO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5976475529728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚459024420771328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270159048704 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103625888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420519074850816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5975819167232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134729665 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34061157 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5975819167232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 J5ƒbatch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238375591 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60455660 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134749703 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403697768170496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414542581318144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚420519074850816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34060716 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚459024612529664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788410425314927e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5976475529728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94516819 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6270326866944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5528.302610457 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134729665 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60455662 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5953823944102 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414542599321088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103625888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134749703 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452754261722624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420518400485376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103625886 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409651575075238 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403697751131136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94516376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5954478381858 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚420518400485376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103625886 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452754285662720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0È6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238388021 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6271014211072 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5977294970880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚7149 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5976638595072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5977294970880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5976638595072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420576247330816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238388021 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414598934356992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452815912460288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 5528.99127769 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238408087 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94529101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚452815936400384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34065273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103639902 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94529564 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚420576247330816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103639902 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚420575572952064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134768183 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459086926671360 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚585Ÿa¾ó&²íM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚420575572952064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134768183 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459086926671360 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚585 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238388021 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6271014211072 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5977294970880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚7149 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5976638595072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5977294970880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5976638595072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420576247330816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238388021 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414598934356992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452815912460288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 5528.99127769 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238408087 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94529101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚452815936400384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34065273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103639902 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94529564 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚420576247330816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103639902 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414ÉH;üN2³- _wandbruntime‚5528  _runtime‚5528Ûôtô\´Ö /loss/FRt3/assignment_loss‚1.525063617927272 2loss/SRqqt3/assignment_loss‚0.8979661017966767 1loss/FBt2/assignment_loss‚0.16688413477303193 1loss/FBt3/assignment_loss‚0.12037286954755246 0loss/FBt4/assignment_loss‚0.2807512810250046 .loss/FRt1/detection_loss‚2.949330849894162 -loss/FBt2/detection_loss‚2.49825462273189 epoch‚5 2loss/SRqqt2/assignment_loss‚1.1849176875074152 1loss/SRqqt2/detection_loss‚1.6558693084451888 _runtime‚5528.992659025 2loss/FBt1/assignment_loss‚0.010676244464279576 /loss/FBt4/detection_loss‚3.3631656765937805 /loss/FRt4/assignment_loss‚2.143049029927505 /loss/FRt2/detection_loss‚2.3153026439249516 .loss/FBt1/detection_loss‚2.768349224870855 .loss/FBt3/detection_loss‚0.558394326669672 % _timestamp‚1.7884104260052972e+09 _step‚586 0loss/SRqqt1/detection_loss‚2.780684410379483 1loss/SRqqt3/detection_loss‚1.0697546296442548 0loss/FRt2/assignment_loss‚2.9500767961144447 2loss/SRqqt1/assignment_loss‚1.6352978818691695 /loss/FRt3/detection_loss‚1.9151026349423863 1loss/SRqqt4/detection_loss‚1.6788531833368798 0loss/FRt1/assignment_loss‚3.4800335251051804 2loss/SRqqt4/assignment_loss‚1.1368072929427675 /loss/FRt4/detection_loss‚2.5789127608663156 trainer/global_step‚7149 &loss/total_loss‚41.663870735300115•MraµÛ _runtime‚5528.992659025 2loss/FBt1/assignment_loss‚0.010676244464279576 /loss/FBt4/detection_loss‚3.3631656765937805 /loss/FRt4/assignment_loss‚2.143049029927505 /loss/FRt2/detection_loss‚2.3153026439249516 .loss/FBt1/detection_loss‚2.768349224870855 .loss/FBt3/detection_loss‚0.558394326669672 % _timestamp‚1.7884104260052972e+09 _step‚586 0loss/SRqqt1/detection_loss‚2.780684410379483 1loss/SRqqt3/detection_loss‚1.0697546296442548 0loss/FRt2/assignment_loss‚2.9500767961144447 2loss/SRqqt1/assignment_loss‚1.6352978818691695 /loss/FRt3/detection_loss‚1.9151026349423863 1loss/SRqqt4/detection_loss‚1.6788531833368798 0loss/FRt1/assignment_loss‚3.4800335251051804 2loss/SRqqt4/assignment_loss‚1.1368072929427675 /loss/FRt4/detection_loss‚2.5789127608663156 trainer/global_step‚7149 &loss/total_loss‚41.663870735300115 /loss/FRt3/assignment_loss‚1.525063617927272 2loss/SRqqt3/assignment_loss‚0.8979661017966767 1loss/FBt2/assignment_loss‚0.16688413477303193 1loss/FBt3/assignment_loss‚0.12037286954755246 0loss/FBt4/assignment_loss‚0.2807512810250046 .loss/FRt1/detection_loss‚2.949330849894162 -loss/FBt2/detection_loss‚2.49825462273189 epoch‚5 2loss/SRqqt2/assignment_loss‚1.1849176875074152 1loss/SRqqt2/detection_loss‚1.6558693084451888ÊeÀ•¶‚(:’ ÀôãÔþß« gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.611' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.322200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø¦}·‚(:ò ÀôãÔÛÝ‘ & disk.nvme0n1.out‚39378.63818359375# disk.dm-0.out‚39378.63818359375 disk.dm-1.out‚ 267.12109375 disk.drbd2265.out‚0% disk./.usageGB‚247.24932861328125 disk.sda2.in‚0( disk.nvme0c0n1.out‚39378.63818359375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1167.703125 disk.dm-0.in‚ 1167.703125 disk.drbd2080.in‚0) disk./.usagePercent‚16.59310989924199! disk.nvme0c0n1.in‚ 1167.703125 disk.drbd1716.in‚0 disk.rbd1.out‚ 26.369140625 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 267.12109375$ memory_percent‚93.86723041534424) proc.memory.availableMB‚ 1004.79296875* proc.memory.percent‚17.107343673706055 cpu‚25.321350767077394 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 267.12109375 disk.sda1.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2802.8671875 network.sent‚ 1.6752107e+07 network.recv‚ 2.042029e+06 disk.dm-1.in‚ 0.1953125 proc.cpu.threads‚17z"’R¸‚(:… ÏôãÔ⸠gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)g2Æy¹‚(:î ÏôãÔš¥‰  disk.drbd2265.out‚0 disk.drbd2080.in‚0* disk./.usagePercent‚16.595728088294663 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1004.53515625 cpu‚25.320621323896333 network.sent‚ 1.6860801e+07 disk.rbd1.out‚ 26.369140625 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 267.54296875 network.recv‚ 2.04839e+06 disk.drbd1716.out‚0 disk.sda.out‚ 267.54296875 disk.rbd0.in‚0* proc.memory.percent‚17.107343673706055$ disk./.usageGB‚247.2883415222168! disk.nvme0c0n1.in‚ 1167.703125' disk.nvme0c0n1.out‚39508.8037109375 disk.dm-1.out‚ 267.54296875 disk.drbd2265.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0" disk.dm-0.out‚39508.8037109375 disk.sda3.in‚ 0.1953125 disk.drbd2080.out‚0 disk.sda2.in‚0$ memory_percent‚93.86880397796631" proc.memory.rssMB‚ 2802.8671875 disk.nvme0n1.in‚ 1167.703125% disk.nvme0n1.out‚39508.8037109375 disk.dm-0.in‚ 1167.703125 disk.drbd2266.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚02b‘³º~‚^Ù.- _wandbruntime‚5564  _runtime‚5564Üá0kɻà _runtime‚5564.614213456 _step‚587 ' lr-AdamW/pg1‚0.00011580145022060498 ' lr-AdamW/pg2‚0.00011580145022060498 trainer/global_step‚7199 % _timestamp‚1.7884104616267948e+09Zd2Î¼È ' lr-AdamW/pg1‚0.00011580145022060498 ' lr-AdamW/pg2‚0.00011580145022060498 trainer/global_step‚7199 % _timestamp‚1.7884104616267948e+09 _runtime‚5564.614213456 _step‚587ËÁ<Ö2½- _wandbruntime‚5564  _runtime‚5564q¤gå'¾ßO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104326686 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6016804973056 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423377038233600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6313414849536 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60864062 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423377038233600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412438288419762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _step‚588 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423377712599040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135655717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34288937 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6016804973056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6017461335552 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135675755 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6313238642688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406442974698496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95153449 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417360251263488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104326686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34289387 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462150237395456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417360233260544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95152997 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5994659283510 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚455836822545920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884104616275134e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417360251263488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104326688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚5564.615182904 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240002443 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417360233260544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423377038233600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412438288419762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _step‚588 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423377712599040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135655717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34288937 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6016804973056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6017461335552 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135675755 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6313238642688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406442974698496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95153449 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417360251263488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104326686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34289387 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462150237395456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417360233260544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95152997 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5994659283510 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚455836822545920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884104616275134e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417360251263488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104326688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚5564.615182904Ì0•þCÀ‚(:’ ÞôãÔô†‰ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.269' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.653800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«vÈvÁ‚(:ë ÞôãÔÒüå  disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda2.out‚0* disk./.usagePercent‚16.598879285033625 disk.drbd1716.out‚0 disk.rbd1.out‚ 26.4951171875 disk.drbd2266.out‚0$ memory_percent‚93.86749267578125 cpu‚25.319888413186156 network.sent‚ 1.6866643e+07% disk./.usageGB‚247.33529663085938 disk.drbd1716.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚39669.6943359375" disk.dm-0.out‚39669.6943359375 disk.sda3.out‚ 268.16015625* proc.memory.percent‚17.107343673706055 network.recv‚ 2.052937e+06 disk.nvme0n1.in‚ 1167.703125 disk.dm-1.out‚ 268.16015625 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚39669.6943359375 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 1167.703125 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1167.703125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 268.16015625 disk.drbd2266.in‚0# proc.memory.availableMB‚1004.75" proc.memory.rssMB‚ 2802.8671875 proc.cpu.threads‚17 disk.sda1.in‚0¶ 2Â- _wandbruntime‚5565  _runtime‚5565tlªï&ÃéM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135694235 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884104623121555e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104340704 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6314119564800 _runtime‚5565.300309372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6017624400896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚455898473283584 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104340702 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462212748070400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚406497861789696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135674171 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6314274786816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60872228 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135694235 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240014873 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5995475718474 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60872230 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104340704 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6017624400896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417416586299392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6018280776704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95166106 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95165660 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406497878829056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417416604302336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417416586299392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34293876 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412493337508170 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 trainer/global_step‚7199 _step‚589 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423434210700288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5996130156438 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423434885079040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240034939 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34293432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_endwq§Œ;&/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34293432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚455898473283584 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104340702 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462212748070400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚406497861789696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135674171 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6314274786816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60872228 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135694235 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240014873 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5995475718474 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60872230 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104340704 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6017624400896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417416586299392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6018280776704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95166106 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95165660 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406497878829056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417416604302336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417416586299392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34293876 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412493337508170 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 trainer/global_step‚7199 _step‚589 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423434210700288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5996130156438 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423434885079040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324Íþwf2Å- _wandbruntime‚5565  _runtime‚5565bšwŠYÆÓ /loss/FRt2/assignment_loss‚2.530454093659366 0loss/SRqqt4/detection_loss‚1.636255637609533 /loss/FBt1/detection_loss‚2.5472816015992845 2loss/SRqqt1/assignment_loss‚1.8517913934633587 2loss/SRqqt4/assignment_loss‚1.0720391401222773 1loss/FBt1/assignment_loss‚0.38171841831146075 0loss/FBt2/assignment_loss‚0.4236303408231054 /loss/FRt1/detection_loss‚2.7820225656032562 1loss/SRqqt2/detection_loss‚1.5331124896710773 0loss/FRt3/assignment_loss‚1.3717159665360743 .loss/FRt2/detection_loss‚1.974748001063258 .loss/FRt4/detection_loss‚2.037555175045362 /loss/SRqqt1/detection_loss‚2.53982655117006 /loss/FRt1/assignment_loss‚3.780463500155343 .loss/FRt3/detection_loss‚1.735607823318973 /loss/FBt3/detection_loss‚0.7724055939793794 .loss/FBt4/detection_loss‚2.071403159035577 epoch‚5 _step‚590 2loss/SRqqt2/assignment_loss‚1.2219086531901135 1loss/FBt3/assignment_loss‚0.14647140720655352 &loss/total_loss‚39.436874272574386 0loss/FRt4/assignment_loss‚2.5466841699985356 1loss/SRqqt3/assignment_loss‚1.141312333282785 1loss/FBt4/assignment_loss‚0.08667041639111833 1loss/SRqqt3/detection_loss‚0.9142378613460755 /loss/FBt2/detection_loss‚2.3375579799924577 % _timestamp‚1.7884104623141797e+09 _runtime‚5565.301541496 trainer/global_step‚7199R"ú«^ÇØ /loss/FRt1/detection_loss‚2.7820225656032562 1loss/SRqqt2/detection_loss‚1.5331124896710773 0loss/FRt3/assignment_loss‚1.3717159665360743 .loss/FRt2/detection_loss‚1.974748001063258 .loss/FRt4/detection_loss‚2.037555175045362 /loss/SRqqt1/detection_loss‚2.53982655117006 /loss/FRt1/assignment_loss‚3.780463500155343 .loss/FRt3/detection_loss‚1.735607823318973 /loss/FBt3/detection_loss‚0.7724055939793794 .loss/FBt4/detection_loss‚2.071403159035577 epoch‚5 _step‚590 2loss/SRqqt2/assignment_loss‚1.2219086531901135 1loss/FBt3/assignment_loss‚0.14647140720655352 &loss/total_loss‚39.436874272574386 0loss/FRt4/assignment_loss‚2.5466841699985356 1loss/SRqqt3/assignment_loss‚1.141312333282785 1loss/FBt4/assignment_loss‚0.08667041639111833 1loss/SRqqt3/detection_loss‚0.9142378613460755 /loss/FBt2/detection_loss‚2.3375579799924577 % _timestamp‚1.7884104623141797e+09 _runtime‚5565.301541496 trainer/global_step‚7199 /loss/FRt2/assignment_loss‚2.530454093659366 0loss/SRqqt4/detection_loss‚1.636255637609533 /loss/FBt1/detection_loss‚2.5472816015992845 2loss/SRqqt1/assignment_loss‚1.8517913934633587 2loss/SRqqt4/assignment_loss‚1.0720391401222773 1loss/FBt1/assignment_loss‚0.38171841831146075 0loss/FBt2/assignment_loss‚0.4236303408231054Îûë¸Ä’È‚(:‡ íôãÔljï gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.651' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~%‚+|É‚(:ñ íôãÔ¿Øó  disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0$ memory_percent‚93.86768341064453 network.sent‚ 1.6964254e+07 network.recv‚ 2.059106e+06 disk.dm-0.in‚ 1167.703125 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 268.62109375 disk.sda1.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚17.107343673706055 disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17% disk./.usageGB‚247.38350677490234 disk.sda1.in‚0& proc.memory.availableMB‚ 1004.71875 disk.nvme0n1.in‚ 1167.703125& disk.nvme0n1.out‚39755.38623046875# disk.dm-0.out‚39755.38623046875 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 1167.703125* disk./.usagePercent‚16.602114708251356 disk.dm-1.out‚ 268.62109375 disk.rbd1.out‚ 26.4951171875 cpu‚25.319174641135522 disk.sda.out‚ 268.62109375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚39755.38623046875" proc.memory.rssMB‚ 2802.8671875°’Ê‚(:‡ üôãÔèªñ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.038' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05«a tË‚(:é üôãÔñèÑ  disk.drbd2266.in‚0 disk./.usageGB‚ 247.41015625 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 1167.703125$ memory_percent‚93.86770725250244" proc.memory.rssMB‚ 2802.8671875 network.recv‚ 2.064659e+06 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚39881.66357421875) disk./.usagePercent‚16.60390317688556 disk.sda.out‚ 269.1328125 disk.sda3.out‚ 269.1328125 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 26.5576171875 disk.drbd2080.in‚0# disk.dm-0.out‚39881.66357421875 disk.dm-0.in‚ 1167.703125 network.sent‚ 1.706185e+07 disk.sda1.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1167.703125 disk.drbd2266.out‚0 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 269.1328125 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚39881.66357421875) proc.memory.availableMB‚ 1004.71484375* proc.memory.percent‚17.107343673706055 cpu‚25.318449210653103 proc.cpu.threads‚17ê«ä2Ì- _wandbruntime‚5601  _runtime‚56018×ÈÍ trainer/global_step‚7249 $ _timestamp‚1.788410498277368e+09 _runtime‚5601.264900447 _step‚591 ' lr-AdamW/pg1‚0.00011384332162733647 ' lr-AdamW/pg2‚0.00011384332162733647}/ÃÐÍÎÇ ' lr-AdamW/pg1‚0.00011384332162733647 ' lr-AdamW/pg2‚0.00011384332162733647 trainer/global_step‚7249 $ _timestamp‚1.788410498277368e+09 _runtime‚5601.264900447 _step‚591Ϟܙ®2Ï- _wandbruntime‚5601  _runtime‚5601&…¦‚ä'ÐÞO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105027488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426235675981824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚420177903205888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚420177885202944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426235675981824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6057790778880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105027486 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136581769 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61272462 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105027488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95789116 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601807 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚592 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105027486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458919359429120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409188181226496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34516216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415223658810054 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6036149060674 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5601.265879934 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136581769 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465275845859840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6058447141376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241629295 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426236350347264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6035494622918 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚420177885202944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6058447141376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34516654 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465275664587264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136601807 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6057790778880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6356486430720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚420177903205888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241609255 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409188164187136Õ¼ot‰ÑãO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6356486430720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚420177903205888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241609255 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409188164187136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105027488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426235675981824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚420177903205888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚420177885202944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426235675981824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6057790778880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105027486 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136581769 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61272462 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105027488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95789116 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601807 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚592 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105027486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458919359429120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409188181226496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34516216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415223658810054 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_ö¤Ã`pool.allocated‚ 6036149060674 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5601.265879934 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136581769 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465275845859840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6058447141376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241629295 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426236350347264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6035494622918 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚420177885202944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6058447141376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34516654 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465275664587264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136601807 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6057790778880 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6357352033792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420234256244736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105041502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415280050852902 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95801272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105041502 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6036311057882 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136620287 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136620287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61280628 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788410498958859e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6058610206720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105041504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136600223 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6357182131712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458981010166784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚420234238241792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6059266582528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105041504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465338362200576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465338168358400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415279379375578 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409243068317696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136600223 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426292848448512 _step‚593 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241641725 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34521084 _runtime‚5601.946916549 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426293522827264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6059266582528SÉ¿ëó&ÔíM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420234256244736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105041502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415280050852902 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95801272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105041502 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6036311057882 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136620287 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136620287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61280628 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788410498958859e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6058610206720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105041504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136600223 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6357182131712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458981010166784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚420234238241792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6059266582528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105041504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465338362200576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465338168358400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415279379375578 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409243068317696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136600223 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426292848448512 _step‚593 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241641725 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34521084 _runtime‚5601.946916549 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426293522827264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6059266582528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚420234256244736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6058610206720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6357352033792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Ñš5R2Õ- _wandbruntime‚5601  _runtime‚5601nP3Þ[ÖÕ .loss/FBt2/detection_loss‚2.587253898382187 /loss/FRt2/detection_loss‚2.4145264002409856 /loss/FRt1/assignment_loss‚3.126718919724226 2loss/SRqqt4/assignment_loss‚1.4736162184778114 1loss/FBt4/assignment_loss‚0.30795013050131104 1loss/SRqqt1/detection_loss‚2.3969220792925037 0loss/FRt2/assignment_loss‚2.8117399134419183 2loss/SRqqt2/assignment_loss‚1.3404815830290318 /loss/FRt1/detection_loss‚2.7196694407612085 .loss/FRt3/detection_loss‚1.880695600145393 .loss/FBt4/detection_loss‚2.464251384139061 epoch‚5 0loss/FRt4/assignment_loss‚2.9514929574468862 2loss/SRqqt3/assignment_loss‚0.9253407579592683 0loss/FBt1/assignment_loss‚0.8048279744980391 0loss/FBt2/assignment_loss‚0.4829687498083022 _step‚594 0loss/SRqqt2/detection_loss‚1.746749915755712 /loss/FBt3/detection_loss‚0.5219168413824838 %loss/total_loss‚41.52593659786142 2loss/SRqqt1/assignment_loss‚1.4937057334023554 1loss/SRqqt3/detection_loss‚0.9431413895817418 1loss/SRqqt4/detection_loss‚1.9964241667286209 /loss/FRt4/detection_loss‚2.5552895247288374 .loss/FBt1/detection_loss‚2.328840419650078 trainer/global_step‚7249 _runtime‚5601.948253109 % _timestamp‚1.7884104989608982e+09 0loss/FRt3/assignment_loss‚1.1583844701449075 1loss/FBt3/assignment_loss‚0.09302812863854937´û`×Ú 0loss/FRt3/assignment_loss‚1.1583844701449075 1loss/FBt3/assignment_loss‚0.09302812863854937 .loss/FBt2/detection_loss‚2.587253898382187 /loss/FRt2/detection_loss‚2.4145264002409856 /loss/FRt1/assignment_loss‚3.126718919724226 2loss/SRqqt4/assignment_loss‚1.4736162184778114 1loss/FBt4/assignment_loss‚0.30795013050131104 1loss/SRqqt1/detection_loss‚2.3969220792925037 0loss/FRt2/assignment_loss‚2.8117399134419183 2loss/SRqqt2/assignment_loss‚1.3404815830290318 /loss/FRt1/detection_loss‚2.7196694407612085 .loss/FRt3/detection_loss‚1.880695600145393 .loss/FBt4/detection_loss‚2.464251384139061 epoch‚5 0loss/FRt4/assignment_loss‚2.9514929574468862 2loss/SRqqt3/assignment_loss‚0.9253407579592683 0loss/FBt1/assignment_loss‚0.8048279744980391 0loss/FBt2/assignment_loss‚0.4829687498083022 _step‚594 0loss/SRqqt2/detection_loss‚1.746749915755712 /loss/FBt3/detection_loss‚0.5219168413824838 %loss/total_loss‚41.52593659786142 2loss/SRqqt1/assignment_loss‚1.4937057334023554 1loss/SRqqt3/detection_loss‚0.9431413895817418 1loss/SRqqt4/detection_loss‚1.9964241667286209 /loss/FRt4/detection_loss‚2.5552895247288374 .loss/FBt1/detection_loss‚2.328840419650078 trainer/global_step‚7249 _runtime‚5601.948253109 % _timestamp‚1.7884104989608982e+09ÒÛhvH’Ø‚(:‡ ‹õãÔ‘‰ë gpu.0.gpu‚12.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Œó-pÙ‚(:å ‹õãÔßô‘ $ disk./.usageGB‚247.4517059326172 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 1167.703125* disk./.usagePercent‚16.606691610948506 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 269.53125# memory_percent‚93.8678503036499 proc.cpu.threads‚17 disk.dm-1.out‚ 269.53125 disk.drbd2080.in‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚39978.5283203125* proc.memory.percent‚17.107343673706055 disk.dm-0.in‚ 1167.703125" proc.memory.rssMB‚ 2802.8671875 disk.nvme0n1.in‚ 1167.703125 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.rbd1.in‚0 cpu‚25.32178583953746 disk.rbd0.in‚0 disk.sda1.out‚0 network.sent‚ 1.7067879e+07% disk.nvme0n1.out‚39978.5283203125" disk.dm-0.out‚39978.5283203125 disk.sda.out‚ 269.53125 disk.drbd2266.out‚0) proc.memory.availableMB‚ 1004.69140625 network.recv‚ 2.068852e+06 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 26.5576171875 disk.sda3.in‚ 0.1953125ñŒÄU’Ú‚(:‡ šõãÔÉ…ÿ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SQísvÛ‚(:ë šõãÔ¶×à  disk.sda1.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚40053.1064453125 disk.rbd1.in‚0" proc.memory.rssMB‚ 2802.8671875 network.recv‚ 2.075263e+06 disk.dm-1.out‚ 270.19921875 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚40053.1064453125 disk.dm-0.in‚ 1167.703125 disk.dm-1.in‚ 0.1953125 disk.drbd1716.out‚0 network.sent‚ 1.7176406e+07$ disk./.usageGB‚247.4710693359375 disk.sda3.out‚ 270.19921875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 26.68359375) disk./.usagePercent‚16.60799110519231 disk.sda3.in‚ 0.1953125 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 1167.703125( proc.memory.availableMB‚ 1004.6796875 cpu‚25.33450048454891 disk.nvme0n1.in‚ 1167.703125' disk.nvme0c0n1.out‚40053.1064453125 disk.drbd1716.in‚0$ memory_percent‚93.86792182922363 disk.drbd2265.out‚0 disk.sda.out‚ 270.19921875 disk.sda1.in‚0* proc.memory.percent‚17.107343673706055ã!‘Ü‚(:† ©õãԕŘ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚177.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KK+VwÝ‚(:ì ©õãÔµ”£ % disk./.usageGB‚247.50344467163086! disk.nvme0c0n1.in‚ 1167.703125 disk.rbd1.out‚ 26.76953125 disk.dm-0.in‚ 1167.703125* proc.memory.percent‚17.107343673706055* disk./.usagePercent‚16.610163841135407 disk.sda2.out‚0 disk.nvme0n1.in‚ 1167.703125 disk.drbd2265.out‚0 disk.sda.out‚ 270.734375" proc.memory.rssMB‚ 2802.8671875 cpu‚25.346338097861235 network.sent‚ 1.7182698e+07 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 270.734375 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 270.734375 network.recv‚ 2.079771e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚40155.83349609375& disk.nvme0n1.out‚40155.83349609375# disk.dm-0.out‚40155.83349609375 disk.sda3.in‚ 0.1953125 disk.drbd2080.out‚0$ memory_percent‚93.86789798736572) proc.memory.availableMB‚ 1004.68359375 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125à@Š2Þ- _wandbruntime‚5652  _runtime‚5652œDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6080448653312 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚96393369 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6079955823104 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚4¤wX%17249650888186 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚428344788464640 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚61631996 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚467462387042304 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6080448653312 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 105627062 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 137034830 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚461083500119040 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34761524 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚467462245661184 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 _runtime‚5652.585110797 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 137034830 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚428344277631488 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 trainer/global_step‚7253 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚461083524059136 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 117441024 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 242677869 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 141381120 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 242677869 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 105627060 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 _step‚595 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788410549597298e+09 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 105627062 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚96393522 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6080448653312 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚96393369 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6079955823104 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚417249650888186 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚428344788464640 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚61631996 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚467462387042304 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6080448653312 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 105627062 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 137034830 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚461083500119040 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34761524 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚467462245661184 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 _runtime‚5652.585110797Ó…Ö¤2á- _wandbruntime‚5653  _runtime‚5653,DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242679998 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 105637866 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 _runtime‚5653.517232711 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6379110668800 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6080813779968 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 137058109 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚96404272 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 117441024 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6080320949760 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚61638500 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6058426357138 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚422301169840128 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 105637866 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 141381120 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚467500486521344 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚417285816210834 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242679998 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 105637866 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 _runtime‚5653.517232711 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6379110668800 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6080813779968 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 137058109 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚96404272 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 117441024 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6080320949760 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚61638500 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6058426357138 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚422301169840128 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 105637866 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 141381120 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚467500486521344 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚417285816210834 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841405153274536 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004270324949175119 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4197048246860504 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4045409858226776 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16099141538143158 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.36515146493911743 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14242881536483765 particle/accuracy_1_of_2‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.587327241897583 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.034958429634571075 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4592069983482361 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_6‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3841414749622345 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.25989392399787903 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12749721109867096 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_4_of_6‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5694481134414673 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4230020344257355 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9429019689559937 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4551536738872528 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9867965579032898 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 jet/accuracy_6_of_7‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3493705987930298 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5414614081382751 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9830870032310486 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4418860077857971 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.353218674659729 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5486077666282654 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5262947082519531 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.031072234734892845 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5018278360366821 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5554306507110596 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.383442223072052 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5056227445602417 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 $ _timestamp‚1.788410562932464e+09 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15471692383289337 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4451718330383301 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900845885276794 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,jet/accuracy_3_of_3‚0.027180781587958336 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3867102265357971 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9850526452064514 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3672791123390198 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9736841917037964 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1928098350763321 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13829787075519562 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6051587462425232 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3300595283508301 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.027645407244563103 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5237622261047363 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22030095756053925 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.07662230730056763 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3125 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15852101147174835 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2639276683330536 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1418052315711975 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17180147767066956 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909886121749878 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06380332261323929 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878789007663727 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20675216615200043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0769021064043045 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.262037068605423 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_2_of_7‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5946289896965027 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21759261190891266 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3101598918437958 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14888888597488403 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893277883529663 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989040493965149 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39534884691238403 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3813389241695404 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999630868434906 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.019736841320991516 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.23076923191547394 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41788068413734436 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.484255313873291 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4745405316352844 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006330822594463825 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17104628682136536 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.47460106015205383 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9479166865348816 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5632205009460449 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5841781497001648 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22076092660427094 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5892360806465149 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07239172607660294 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4551282227039337 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1234322115778923 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47191983461380005 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.996454119682312 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.581065833568573 particle/accuracy_2_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5987179279327393 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5884204506874084 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45644527673721313 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1948584020137787 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9905285835266113 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1548904925584793 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4698185920715332 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13971252739429474 +jet/accuracy_3_of_5‚0.11500256508588791 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20317456126213074 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_6_of_7‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012736205942928791 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4222223162651062 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09608306735754013 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 epoch‚5 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_4‚0.3712914288043976 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 jet/accuracy_1_of_7‚1 jet/accuracy_3_of_7‚0 particle/accuracy_2_of_7‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015304669737815857 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4593574106693268 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20673075318336487 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21149426698684692 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4279017448425293 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24541231989860535 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5621246695518494 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9910542368888855 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4543103575706482 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013403880409896374 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24209152162075043 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3672087490558624 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.34907400608062744 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚42.29020107222?Hͼ©<319 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984968304634094 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 +jet/accuracy_4_of_6‚0.03846153989434242 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9814196825027466 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 particle/accuracy_3_of_7‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4547920227050781 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3641975522041321 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29685598611831665 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.013565892353653908 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8574321269989014 particle/accuracy_1_of_3‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45228126645088196 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.156779944896698 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26378127932548523 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13555724918842316 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1928059160709381 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028481582179665565 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 )jet/accuracy_2_of_5‚0.495135635137558 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9896368980407715 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6221076846122742 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.551468551158905 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 particle/accuracy_7_of_7‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24849900603294373 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.596130907535553 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6041666865348816 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6041666865348816 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04556650295853615 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4918324947357178 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5324074625968933 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.507447361946106 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915238618850708 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030495433136820793 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6929824948310852 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 (particle/accuracy‚0.8526941537857056 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4479166865348816 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9907498359680176 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16079677641391754 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4135645925998688 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4717840850353241 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12962962687015533 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 trainer/global_step‚7253 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20128150284290314 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14942529797554016 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5237361788749695 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18321965634822845 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 _step‚597 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5191707611083984 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3203752636909485 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45919638872146606 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4551536738872528 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.347222238779068 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.347222238779068 jet/accuracy_4_of_7‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16351227462291718 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9898695945739746 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5161784887313843 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14670692384243011 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 (jet/accuracy_1_of_2‚0.73771071434021 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4873949885368347 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 _runtime‚5665.922121296 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5263701677322388 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1790851503610611 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22701579332351685 ,jet/accuracy_4_of_5‚0.016577061265707016 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_7‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15172414481639862 'particle/f_score‚0.4043358862400055 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14561404287815094 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9795918464660645 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6565384268760681 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09345237165689468 particle/accuracy_4_of_4‚1 particle/accuracy_5_of_5‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04904184117913246 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847981929779053 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19576717913150787 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3441353738307953 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18755044043064117 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4133088290691376 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9511726498603821 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6905325055122375 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4933180809020996 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17793624103069305 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.34819817543029785 particle/accuracy_2_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.998085618019104 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0927785336971283 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21350911259651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2727954089641571 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4631974697113037 particle/accuracy_2_of_3‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8115293383598328 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5367774963378906 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.020602919161319733 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4651528000831604 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 *jet/accuracy_1_of_4‚0.9189388155937195 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34826385974884033 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47109630703926086 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26297587156295776 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.215277761220932 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5761494040489197 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16792245209217072 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224Ì-˜üIC螧 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4222223162651062 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09608306735754013 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 epoch‚5 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_4‚0.3712914288043976 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 jet/accuracy_1_of_7‚1 jet/accuracy_3_of_7‚0 particle/accuracy_2_of_7‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015304669737815857 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4593574106693268 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20673075318336487 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21149426698684692 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4279017448425293 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24541231989860535 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5621246695518494 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9910542368888855 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4543103575706482 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013403880409896374 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24209152162075043 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3672087490558624 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.34907400608062744 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚42.29020107222319 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984968304634094 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 +jet/accuracy_4_of_6‚0.03846153989434242 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9814196825027466 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 particle/accuracy_3_of_7‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4547920227050781 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3641975522041321 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29685598611831665 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.013565892353653908 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8574321269989014 particle/accuracy_1_of_3‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45228126645088196 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.156779944896698 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26378127932548523 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13555724918842316 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1928059160709381 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028481582179665565 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 )jet/accuracy_2_of_5‚0.495135635137558 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9896368980407715 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6221076846122742 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.551468551158905 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 particle/accuracy_7_of_7‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24849900603294373 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.596130907535553 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6041666865348816 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6041666865348816 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04556650295853615 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4918324947357178 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5324074625968933 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.507447361946106 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915238618850708 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030495433136820793 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6929824948310852 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 (particle/accuracy‚0.8526941537857056 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4479166865348816 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9907498359680176 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16079677641391754 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4135645925998688 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4717840850353241 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12962962687015533 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 trainer/global_step‚7253 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20128150284290314 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14942529797554016 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5237361788749695 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18321965634822845 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 _step‚597 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5191707611083984 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3203752636909485 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45919638872146606 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4551536738872528 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.347222238779068 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.347222238779068 jet/accuracy_4_of_7‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16351227462291718 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9898695945739746 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5161784887313843 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14670692384243011 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 (jet/accuracy_1_of_2‚0.73771071434021 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4873949885368347 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 _runtime‚5665.922121296 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5263701677322388 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1790851503610611 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22701579332351685 ,jet/accuracy_4_of_5‚0.016577061265707016 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_7‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15172414481639862 'particle/f_score‚0.4043358862400055 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14561404287815094 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9795918464660645 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6565384268760681 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09345237165689468 particle/accuracy_4_of_4‚1 particle/accuracy_5_of_5‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04904184117913246 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847981929779053 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19576717913150787 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3441353738307953 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18755044043064117 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4133088290691376 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9511726498603821 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6905325055122375 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4933180809020996 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17793624103069305 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.34819817543029785 particle/accuracy_2_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.998085618019104 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0927785336971283 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21350911259651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2727954089641571 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4631974697113037 particle/accuracy_2_of_3‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8115293383598328 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5367774963378906 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.020602919161319733 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4651528000831604 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 *jet/accuracy_1öÞò\P_of_4‚0.9189388155937195 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34826385974884033 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47109630703926086 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26297587156295776 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.215277761220932 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5761494040489197 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16792245209217072 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2891414165496826 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021743958815932274 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.30405405163764954 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1637675166130066 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3199917674064636 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677671432495 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2213938683271408 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_2_of_2‚0.11016011983156204 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4428439736366272 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011051344685256481 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 6validation_average_jet_accuracy‚0.4251892864704132 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16792245209217072 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4609387218952179 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2403974086046219 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841405153274536 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004270324949175119 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4197048246860504 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4045409858226776 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16099141538143158 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.36515146493911743 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14242881536483765 particle/accuracy_1_of_2‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.587327241897583 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.034958429634571075 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4592069983482361 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_6‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3841414749622345 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.25989392399787903 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12749721109867096 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_4_of_6‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5694481134414673 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4230020344257355 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9429019689559937 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4551536738872528 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9867965579032898 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 jet/accuracy_6_of_7‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3493705987930298 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5414614081382751 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9830870032310486 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4418860077857971 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.353218674659729 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5486077666282654 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5262947082519531 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.031072234734892845 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5018278360366821 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5554306507110596 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.383442223072052 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5056227445602417 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 $ _timestamp‚1.788410562932464e+09 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15471692383289337 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4451718330383301 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900845885276794 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,jet/accuracy_3_of_3‚0.027180781587958336 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3867102265357971 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9850526452064514 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3672791123390198 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9736841917037964 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1928098350763321 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13829787075519562 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6051587462425232 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3300595283508301 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.027645407244563103 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5237622261047363 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22030095756053925 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.07662230730056763 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3125 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15852101147174835 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2639276683330536 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1418052315711975 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17180147767066956 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909886121749878 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06380332261323929 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878789007663727 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20675216615200043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0769021064043045 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.262037068605423 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_2_of_7‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5946289896965027 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21759261190891266 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3101598918437958 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14888888597488403 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893277883529663 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989040493965149 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39534884691238403 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3813389241695404 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999630868434906 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.019736841320991516 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.23076923191547394 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41788068413734436 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.484255313873291 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4745405316352844 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006330822594463825 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17104628682136536 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.47460106015205383 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9479166865348816 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5632205009460449 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5841781497001648 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22076092660427094 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5892360806465149 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07239172607660294 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4551282227039337 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1234322115778923 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47191983461380005 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.996454119682312 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.581065833568573 particle/accuracy_2_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5987179279327393 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5884204506874084 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45644527673721313 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1948584020137787 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9905285835266113 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1548904925584793 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4698185920715332 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13971252739429474 +jet/accuracy_3_of_5‚0.11500256508588791 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20317456126213074 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_6_of_7‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012736205942928791Õâ´ò"é‚(:’ ÇõãÔÐæ— gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚185.937' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.187400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”q$sê‚(:è ÇõãÔ¦—ï  disk.drbd2265.in‚0 disk.sda.out‚ 272.171875 disk.drbd2080.out‚0 disk.rbd1.out‚ 26.83203125 cpu‚25.366579652281086 disk.sda1.in‚0 disk.dm-1.out‚ 272.171875 disk.dm-1.in‚ 0.1953125 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 1167.703125$ disk./.usageGB‚247.5586166381836 disk.sda.in‚ 0.1953125* proc.memory.percent‚17.101335525512695* disk./.usagePercent‚16.613866478103123 disk.sda3.out‚ 272.171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚40318.5322265625 disk.dm-0.in‚ 1167.703125 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 1167.703125' disk.nvme0c0n1.out‚40318.5322265625" disk.dm-0.out‚40318.5322265625) proc.memory.availableMB‚ 1005.16015625 network.sent‚ 1.7296352e+07 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd1716.in‚0$ memory_percent‚93.86498928070068" proc.memory.rssMB‚ 2801.8828125 proc.cpu.threads‚17 network.recv‚ 2.089809e+06¬µtë‚(:’ ÖõãÔ«ž¡ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚125.542' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.108400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0to£Ãeì‚(:Ú ÖõãÔãžß $ disk./.usageGB‚247.5979232788086 disk.drbd2265.in‚0 disk.sda3.out‚ 272.85546875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0c0n1.out‚ 40410.875 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.out‚ 40410.875 disk.drbd2265.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 1167.703125 network.sent‚ 1.7420757e+07 disk.dm-1.out‚ 272.85546875 disk.sda.out‚ 272.85546875 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.nvme0n1.out‚ 40410.875" proc.memory.rssMB‚ 2801.8828125 disk.sda.in‚ 0.1953125 disk.nvme0n1.in‚ 1167.703125* proc.memory.percent‚17.101335525512695 cpu‚25.365749102471074* disk./.usagePercent‚16.616504379735936 disk.rbd1.out‚ 26.9580078125$ memory_percent‚93.86506080627441( proc.memory.availableMB‚ 1005.1484375 network.recv‚ 2.096209e+06 disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1167.703125 disk.sda1.out‚0 disk.sda2.in‚0³^Ük2í- _wandbruntime‚5698  _runtime‚5698N¡Á Éîà ' lr-AdamW/pg1‚0.00011189012920333003 ' lr-AdamW/pg2‚0.00011189012920333003 trainer/global_step‚7299 % _timestamp‚1.7884105959357371e+09 _runtime‚5698.923172323 _step‚598EpJÎïÈ ' lr-AdamW/pg1‚0.00011189012920333003 ' lr-AdamW/pg2‚0.00011189012920333003 trainer/global_step‚7299 % _timestamp‚1.7884105959357371e+09 _runtime‚5698.923172323 _step‚598ÖLv/2ð- _wandbruntime‚5698  _runtime‚5698rS¹Q!ñßO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420256803502399 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35027704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425315846962688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420256132025283 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚470810780211712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6121796552192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137989817 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106409064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464388043582976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425315828959744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35027277 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431438299877376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431437625511936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137969779 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137989817 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6121796552192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚470810588453376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6422544870400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431437625511936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97117889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚464388067523072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244378843 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425315846962688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6122452914688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97118318 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106409064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464388043582976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425315828959744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35027277 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431438299877376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431437625511936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137969779 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137989817 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6121796552192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚470810588453376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6422544870400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431437625511936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97117889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚464388067523072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244378843 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425315846962688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6122452914688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97118318 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420311852590807 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6123272355840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425372181998592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62098780 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35031765 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6122615980032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137988233 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6100725335331 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425372200001536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35032188 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244431379 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431379 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106423080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431495472357376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6423412560384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137988233 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138008297 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚470873290237440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6123272355840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464449718260736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425372181998592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464449694320640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244411313 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 % _timestamp‚1.7884105966340787e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425372200001536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97130545 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414211781693440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138008297 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6122615980032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470873106881024 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844Õm][ô&õîM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138008297 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚470873290237440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6123272355840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464449718260736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425372181998592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464449694320640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244411313 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 % _timestamp‚1.7884105966340787e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425372200001536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97130545 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414211781693440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138008297 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6122615980032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470873106881024 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244411313 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420311852590807 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6123272355840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425372181998592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62098780 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35031765 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6122615980032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137988233 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6100725335331 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425372200001536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35032188 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244431379 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431379 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106423080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431495472357376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6423412560384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137988233 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048Øt“} 2ö- _wandbruntime‚5699  _runtime‚5699!€[£ ÷Ú &loss/total_loss‚41.536868084977826 2loss/SRqqt4/assignment_loss‚1.4629642581567168 0loss/FBt4/assignment_loss‚0.3790848761758146 0loss/FRt1/assignment_loss‚2.8607610785044155 0loss/FRt3/assignment_loss‚1.4323300388124254 1loss/FBt1/assignment_loss‚0.26625023991800845 1loss/FBt3/assignment_loss‚0.09603299624405846 .loss/FRt3/detection_loss‚1.660955326880018 1loss/SRqqt2/detection_loss‚1.6389233752020769 /loss/FBt1/detection_loss‚1.9526832591403613 % _timestamp‚1.7884105966358705e+09 2loss/SRqqt2/assignment_loss‚1.5645535691375179 /loss/FBt3/detection_loss‚0.7105257082670832 _step‚601 /loss/FBt4/detection_loss‚2.3859853049119315 0loss/FRt2/assignment_loss‚2.8942307111891834 0loss/FRt4/assignment_lþžsFVoss‚2.6570819914340973 2loss/SRqqt3/assignment_loss‚0.9980198292844538 .loss/FRt4/detection_loss‚2.391981409064361 1loss/SRqqt1/detection_loss‚2.6569965737206593 1loss/SRqqt3/detection_loss‚1.0584374310421771 _runtime‚5699.623218233 /loss/FRt2/detection_loss‚2.3149884072217097 epoch‚6 /loss/FRt1/detection_loss‚2.8854069228355703 2loss/SRqqt1/assignment_loss‚2.2953672941241945 1loss/FBt2/assignment_loss‚0.20938349113566712 .loss/FBt2/detection_loss‚2.862791310995817 trainer/global_step‚7299 1loss/SRqqt4/detection_loss‚1.9011326815795015eTïøeøß /loss/FRt1/detection_loss‚2.8854069228355703 2loss/SRqqt1/assignment_loss‚2.2953672941241945 1loss/FBt2/assignment_loss‚0.20938349113566712 .loss/FBt2/detection_loss‚2.862791310995817 trainer/global_step‚7299 1loss/SRqqt4/detection_loss‚1.9011326815795015 &loss/total_loss‚41.536868084977826 2loss/SRqqt4/assignment_loss‚1.4629642581567168 0loss/FBt4/assignment_loss‚0.3790848761758146 0loss/FRt1/assignment_loss‚2.8607610785044155 0loss/FRt3/assignment_loss‚1.4323300388124254 1loss/FBt1/assignment_loss‚0.26625023991800845 1loss/FBt3/assignment_loss‚0.09603299624405846 .loss/FRt3/detection_loss‚1.660955326880018 1loss/SRqqt2/detection_loss‚1.6389233752020769 /loss/FBt1/detection_loss‚1.9526832591403613 % _timestamp‚1.7884105966358705e+09 2loss/SRqqt2/assignment_loss‚1.5645535691375179 /loss/FBt3/detection_loss‚0.7105257082670832 _step‚601 /loss/FBt4/detection_loss‚2.3859853049119315 0loss/FRt2/assignment_loss‚2.8942307111891834 0loss/FRt4/assignment_loss‚2.6570819914340973 2loss/SRqqt3/assignment_loss‚0.9980198292844538 .loss/FRt4/detection_loss‚2.391981409064361 1loss/SRqqt1/detection_loss‚2.6569965737206593 1loss/SRqqt3/detection_loss‚1.0584374310421771 _runtime‚5699.623218233 /loss/FRt2/detection_loss‚2.3149884072217097 epoch‚6ÙÈĆ‘ù‚(:† åõãԞ㈠gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.655' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.731 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t– Áqú‚(:æ åõãÔúã   disk.sda3.in‚ 0.1953125 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 27.0439453125 disk.rbd1.in‚0 disk.sda.in‚ 0.1953125 disk.sda.out‚ 273.28125* proc.memory.percent‚17.101335525512695 disk.drbd2080.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚40479.9560546875" proc.memory.rssMB‚ 2801.8828125* disk./.usagePercent‚16.619136905210528 disk.drbd2266.in‚0" disk.dm-0.out‚40479.9638671875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 1167.703125 disk.dm-1.in‚ 0.1953125 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 1167.703125 disk.drbd1716.out‚0$ memory_percent‚93.86696815490723( proc.memory.availableMB‚ 1004.8359375 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 1167.703125 disk.dm-1.out‚ 273.28125 disk.sda1.out‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 2.100264e+06% disk./.usageGB‚247.63714981079102 disk.sda3.out‚ 273.28125% disk.nvme0n1.out‚40479.9638671875 cpu‚25.36493549440812 network.sent‚ 1.7426932e+07Ê°£H’û‚(:‡ ôõãÔÞªå gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.274' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hú¥}ü‚(:ò ôõãÔ´Û‹ ( disk./.usagePercent‚16.6229588416892 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 network.recv‚ 2.105551e+06 disk.rbd1.out‚ 27.1064453125 disk.nvme0n1.in‚ 1167.703125# disk.dm-0.out‚40559.26318359375 disk.drbd2265.in‚0 disk.sda3.out‚ 273.87890625 disk.drbd2266.in‚0 network.sent‚ 1.7536371e+07 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda.out‚ 273.87890625 disk.drbd1716.out‚0 cpu‚25.364061217508162 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 1167.703125( disk.nvme0c0n1.out‚40559.26318359375 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0$ memory_percent‚93.86546611785889 disk.rbd0.in‚0 disk.rbd0.out‚0% disk./.usageGB‚247.69409942626953 disk.dm-0.in‚ 1167.703125 disk.dm-1.out‚ 273.87890625 disk.drbd2080.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚40559.26318359375 disk.sda.in‚ 0.1953125 disk.sda1.out‚0) proc.memory.availableMB‚ 1005.08203125" proc.memory.rssMB‚ 2801.8828125* proc.memory.percent‚17.101335525512695Ÿ'¥ý‚(:… ƒöãÔÅÔ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž: |þ‚(:ñ ƒöãÔÁÿâ" proc.memory.rssMB‚ 2801.8828125 proc.cpu.threads‚17 network.recv‚ 2.110042e+06 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚40639.23583984375 network.sent‚ 1.7540212e+07 disk.drbd2265.out‚0 disk.sda.out‚ 274.36328125 disk.drbd2080.in‚0! disk.nvme0c0n1.in‚ 1167.703125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1167.703125 disk.dm-0.in‚ 1167.703125 disk.drbd1716.out‚0 disk.drbd2265.in‚0( disk./.usagePercent‚16.6251413059186( proc.memory.availableMB‚ 1005.0859375 disk.rbd0.in‚0 disk.sda1.out‚0% disk./.usageGB‚247.72661972045898& disk.nvme0n1.out‚40639.23583984375# disk.dm-0.out‚40639.23583984375 disk.drbd2266.in‚0* proc.memory.percent‚17.101335525512695 cpu‚25.363211370969818 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 274.36328125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 274.36328125 disk.rbd1.out‚ 27.1064453125 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125$ memory_percent‚93.86544227600098ÏNqÓ2ÿ- _wandbruntime‚5735  _runtime‚5735Ó⌠ɀà ' lr-AdamW/pg1‚0.00010994227999095928 ' lr-AdamW/pg2‚0.00010994227999095928 trainer/global_step‚7349 % _timestamp‚1.7884106326043222e+09 _runtime‚5735.591992922 _step‚602ôÎ[ÎÈ % _timestamp‚1.7884106326043222e+09 _runtime‚5735.591992922 _step‚602 ' lr-AdamW/pg1‚0.00010994227999095928 ' lr-AdamW/pg2‚0.00010994227999095928 trainer/global_step‚7349ÚØ–2–2‚- _wandbruntime‚5735  _runtime‚5735”}7Åå'ƒßO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6140744239567 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6163438720512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428133498905088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416902101130240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97752379 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6465391289856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138895831 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434296263260160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6162782358016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6162782358016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62499012 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35252920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416902084090880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467470604406272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107109864 _runtime‚5735.593014849 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚434296937625600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚428133480902144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62499014 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467470580466176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6140089801811 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473935971756032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6465557010944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107109864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434296263260160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6163438720512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚603 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246005695 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246025735 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428133480902144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138915869 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97751932 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚423042173892691 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107109866K%1íê'„äO % _timestamp‚1.7884106326052196e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473936161417216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434296937625600 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434296263260160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6163438720512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚603 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246005695 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246025735 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428133480902144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138915869 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97751932 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚423042173892691 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107109866 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6140744239567 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6163438720512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428133498905088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416902101130240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97752379 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6465391289856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138895831 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434296263260160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6162782358016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6162782358016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62499012 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35252920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416902084090880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467470604406272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107109864 _runtime‚5735.593014849 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚434296937625600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚428133480902144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62499014 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467470580466176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6140089801811 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473935971756032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6465557010944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107109864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058231 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97764968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467532231203840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6140906236775 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6164258161664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107123880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416956988221440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246038165 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107123880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35257786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138914285 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6163601785856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62507180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35257349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6466422589440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6141560674739 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97764529 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473998494376960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428189851943936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚7349 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6163601785856 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246038165 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428189833940992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6164258161664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62507182 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107123882 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138914285 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473998677733376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138934349 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467532255143936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423097894458215 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416957005260800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107123882FæÚ4ó&‡íM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058231 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97764968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467532231203840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6140906236775 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6164258161664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107123880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416956988221440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246038165 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107123880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35257786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138914285 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6163601785856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62507180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35257349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6466422589440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6141560674739 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97764529 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473998494376960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428189851943936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚7349 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6163601785856 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246038165 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428189833940992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6164258161664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62507182 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107123882 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138914285 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473998677733376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138934349 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467532255143936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423097894458215 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416957005260800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107123882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _runtime‚5736.283871089 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048ÜÑ¿´Ä2ˆ- _wandbruntime‚5736  _runtime‚5736‰0w[‰Õ /loss/FRt1/detection_loss‚2.6933863100252653 .loss/FRt2/detection_loss‚2.567211732631779 epoch‚6 0loss/FRt4/assignment_loss‚2.1463969568816985 2loss/SRqqt3/assignment_loss‚1.0599078215595523 0loss/SRqqt2/detection_loss‚2.305924195051193 0loss/SRqqt4/detection_loss‚1.756103959998914 /loss/FBt3/detection_loss‚0.6979552442956317 /loss/FBt4/detection_loss‚2.5117753062929427 _runtime‚5736.285307768 /loss/FRt2/assignment_loss‚3.084494411945343 2loss/SRqqt1/assignment_loss‚1.4459870958870107 % _timestamp‚1.7884106332979207e+09 0loss/FRt3/assignment_loss‚1.6783850855297513 2loss/SRqqt2/assignment_loss‚1.2405170244829995 0loss/FBt1/assignment_loss‚0.1863641423239772 1loss/FBt2/assignment_loss‚0.13723372204029666 1loss/FBt3/assignment_loss‚0.12163021783199002 1loss/SRqqt1/detection_loss‚2.4385252874909025 1loss/SRqqt3/detection_loss‚1.1047884426000765 1loss/FBt4/assignment_loss‚0.10935357791770782 trainer/global_step‚7349 %loss/total_loss‚41.98533644126333 2loss/SRqqt4/assignment_loss‚1.2565993148434376 /loss/FRt4/detection_loss‚2.3384625972533715 .loss/FBt2/detection_loss‚3.422450831958226 _step‚605 .loss/FRt3/detection_loss‚1.839462135814958 .loss/FBt1/detection_loss‚2.562778115272522 0loss/FRt1/assignment_loss‚3.2796429113337866y=kâ`ŠÚ 0loss/FRt1/assignment_loss‚3.2796429113337866 /loss/FRt1/detection_loss‚2.6933863100252653 .loss/FRt2/detection_loss‚2.567211732631779 epoch‚6 0loss/FRt4/assignment_loss‚2.1463969568816985 2loss/SRqqt3/assignment_loss‚1.0599078215595523 0loss/SRqqt2/detection_loss‚2.305924195051193 0loss/SRqqt4/detection_loss‚1.756103959998914 /loss/FBt3/detection_loss‚0.6979552442956317 /loss/FBt4/detection_loss‚2.5117753062929427 _runtime‚5736.285307768 /loss/FRt2/assignment_loss‚3.084494411945343 2loss/SRqqt1/assignment_loss‚1.4459870958870107 % _timestamp‚1.7884106332979207e+09 0loss/FRt3/assignment_loss‚1.6783850855297513 2loss/SRqqt2/assignment_loss‚1.2405170244829995 0loss/FBt1/assignment_loss‚0.1863641423239772 1loss/FBt2/assignment_loss‚0.13723372204029666 1loss/FBt3/assignment_loss‚0.12163021783199002 1loss/SRqqt1/detection_loss‚2.4385252874909025 1loss/SRqqt3/detection_loss‚1.1047884426000765 1loss/FBt4/assignment_loss‚0.10935357791770782 trainer/global_step‚7349 %loss/total_loss‚41.98533644126333 2loss/SRqqt4/assignment_loss‚1.2565993148434376 /loss/FRt4/detection_loss‚2.3384625972533715 .loss/FBt2/detection_loss‚3.422450831958226 _step‚605 .loss/FRt3/detection_loss‚1.839462135814958 .loss/FBt1/detection_loss‚2.562778115272522ÝmÓ±‹‚(:’ ’öãÔÕó÷ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á2VËtŒ‚(:é ’öãÔøÑü disk.drbd2080.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 network.sent‚ 1.7546048e+07 disk.rbd1.out‚ 27.232421875 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 274.96875 proc.cpu.threads‚17 disk.sda2.out‚0" disk.dm-0.out‚40726.0927734375 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 cpu‚25.362438700174263% disk./.usageGB‚247.76372146606445 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd2265.in‚0 disk.sda3.out‚ 274.96875 network.recv‚ 2.114106e+06 disk.drbd1716.in‚0 disk.dm-0.in‚ 1167.7109375 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1005.05078125" proc.memory.rssMB‚ 2801.8828125$ memory_percent‚93.86565685272217* proc.memory.percent‚17.101335525512695 disk.dm-1.out‚ 274.96875 disk.sda.in‚ 0.1953125) disk./.usagePercent‚16.62763123519662 disk.sda1.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚40726.0927734375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1167.7109375% disk.nvme0n1.out‚40726.0927734375íOÐ’‚(:‡ ¡öãÔÒíŠ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q»÷ÕzŽ‚(:ï ¡öãÔ­ã’  disk.rbd1.out‚ 27.232421875* disk./.usagePercent‚16.630415829146553 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0$ memory_percent‚93.86715888977051 cpu‚25.361574747210344 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.nvme0n1.in‚ 1167.7109375 disk.dm-0.in‚ 1167.7109375( proc.memory.availableMB‚ 1004.8046875 disk.rbd0.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚40796.3193359375 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd1.in‚0 network.sent‚ 1.7655008e+07 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda.out‚ 275.5546875 disk.drbd1716.in‚0" disk.dm-0.out‚40796.3193359375* proc.memory.percent‚17.101335525512695% disk./.usageGB‚247.80521392822266 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 275.5546875 disk.sda3.out‚ 275.5546875% disk.nvme0n1.out‚40796.3193359375" proc.memory.rssMB‚ 2801.8828125 network.recv‚ 2.120621e+06 disk.sda2.in‚0ÕËá2- _wandbruntime‚5772  _runtime‚5772êR¨‹Éà ' lr-AdamW/pg1‚0.00010800017991908062 ' lr-AdamW/pg2‚0.00010800017991908062 trainer/global_step‚7399 % _timestamp‚1.7884106693824513e+09 _runtime‚5772.369867653 _step‚606#æoÉÎ‘È % _timestamp‚1.7884106693824513e+09 _runtime‚5772.369867653 _step‚606 ' lr-AdamW/pg1‚0.00010800017991908062 ' lr-AdamW/pg2‚0.00010800017991908062 trainer/global_step‚7399Þ*¸82’- _wandbruntime‚5772  _runtime‚5772I±?På'“ßO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139821883 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚437155575373824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437154901008384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚430951150847488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6508254700544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139821883 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6203768163840 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6204424526336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚477061372049920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430951150847488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62907412 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107810664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139841921 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425828215760099 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6204424526336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚477061372049920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430951150847488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62907412 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107810664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139841921 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425828215760099 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚437155575373824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437154901008384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚430951150847488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6508254700544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139821883 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6203768163840 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437212073475072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6182396014147 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚608 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419702211788800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6205243967488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6204587591680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98399494 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35483914 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685083 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107824682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6509280330240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5773.066400591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107824682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6509114622464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425884607802947 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139840337 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6205243967488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431007485883392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139860401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107824680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107824680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431007503886336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139860401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6204587591680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6181741576183 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247685083 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437212747853824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477123882709504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470614768087040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884106700786152e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844§BàÆô&—îM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425884607802947 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139840337 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6205243967488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431007485883392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139860401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107824680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107824680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431007503886336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139860401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6204587591680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6181741576183 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247685083 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437212747853824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477123882709504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470614768087040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884106700786152e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35484366 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437212073475072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6182396014147 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚608 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419702211788800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6205243967488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6204587591680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98399494 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35483914 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685083 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107824682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6509280330240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5773.066400591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107824682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6509114622464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0àƒ†~q2˜- _wandbruntime‚5773  _runtime‚5773JH¬ßY™Ó 1loss/SRqqt3/assignment_loss‚1.084523488082649 /loss/FRt4/detection_loss‚2.2832983350381255 /loss/FBt4/detection_loss‚2.2828416228294373 trainer/global_step‚7399 /loss/FRt1/assignment_loss‚2.934512182193644 1loss/SRqqt2/detection_loss‚1.9139152978395306 _step‚609 0loss/FRt3/assignment_loss‚1.6183747439472764 1loss/FBt4/assignment_loss‚0.23018506278458517 %loss/total_loss‚40.14633701663816 epoch‚6 0loss/FRt4/assignment_loss‚2.9026027731597424 1loss/FBt1/assignment_loss‚0.11940214315865721 1loss/SRqqt3/detection_loss‚0.8828264399267295 /loss/FBt3/detection_loss‚0.5876056528986847 2loss/SRqqt2/assignment_loss‚1.2946982607678619 -loss/FBt2/detection_loss‚2.11143473982811 % _timestamp‚1.7884106700805242e+09 .loss/FRt2/assignment_loss‚3.31867320900378 0loss/FBt2/assignment_loss‚0.3980505323561374 /loss/FRt1/detection_loss‚2.6925265420885647 .loss/FBt1/detection_loss‚2.523022258823568 2loss/SRqqt4/assignment_loss‚1.5266201918028177 1loss/SRqqt4/detection_loss‚1.7827995180610627 _runtime‚5773.067892986 1loss/FBt3/assignment_loss‚0.08856696307581464 /loss/FRt2/detection_loss‚2.0432877320313256 /loss/FRt3/detection_loss‚1.7912396138740911 0loss/SRqqt1/detection_loss‚2.062289197777593 1loss/SRqqt1/assignment_loss‚1.673040515288364žÓ|^šØ .loss/FBt1/detection_loss‚2.523022258823568 2loss/SRqqt4/assignment_loss‚1.5266201918028177 1loss/SRqqt4/detection_loss‚1.7827995180610627 _runtime‚5773.067892986 1loss/FBt3/assignment_loss‚0.08856696307581464 /loss/FRt2/detection_loss‚2.0432877320313256 /loss/FRt3/detection_loss‚1.7912396138740911 0loss/SRqqt1/detection_loss‚2.062289197777593 1loss/SRqqt1/assignment_loss‚1.673040515288364 1loss/SRqqt3/assignment_loss‚1.084523488082649 /loss/FRt4/detection_loss‚2.2832983350381255 /loss/FBt4/detection_loss‚2.2828416228294373 trainer/global_step‚7399 /loss/FRt1/assignment_loss‚2.934512182193644 1loss/SRqqt2/detection_loss‚1.9139152978395306 _step‚609 0loss/FRt3/assignment_loss‚1.6183747439472764 1loss/FBt4/assignment_loss‚0.23018506278458517 %loss/total_loss‚40.14633701663816 epoch‚6 0loss/FRt4/assignment_loss‚2.9026027731597424 1loss/FBt1/assignment_loss‚0.11940214315865721 1loss/SRqqt3/detection_loss‚0.8828264399267295 /loss/FBt3/detection_loss‚0.5876056528986847 2loss/SRqqt2/assignment_loss‚1.2946982607678619 -loss/FBt2/detection_loss‚2.11143473982811 % _timestamp‚1.7884106700805242e+09 .loss/FRt2/assignment_loss‚3.31867320900378 0loss/FBt2/assignment_loss‚0.3980505323561374 /loss/FRt1/detection_loss‚2.6925265420885647á~Dòâ’›‚(:‡ °öãÔáø™ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÁèÏœ‚(:ï °öãÔ„¸†  disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125( disk.nvme0c0n1.out‚40876.41943359375' proc.memory.availableMB‚ 1004.328125 disk.rbd0.out‚0# disk.dm-0.out‚40876.41943359375 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 276.2109375 disk.drbd2266.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.101335525512695 proc.cpu.threads‚17 network.sent‚ 1.7660294e+07 disk.rbd1.in‚Ào°ð«0 disk.rbd1.out‚ 27.294921875 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1167.7109375$ memory_percent‚93.87006759643555 network.recv‚ 2.124589e+06 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2080.in‚0 cpu‚25.36081271543134) disk./.usagePercent‚16.63270812060814% disk./.usageGB‚247.83937072753906 disk.sda.out‚ 276.2109375 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 276.2109375 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2801.8828125 disk.drbd1716.out‚0& disk.nvme0n1.out‚40876.41943359375ø\Ä‚(:’ ¿öãÔ”ˆÓ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.861' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.172199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aOÝ|ž‚(:ñ ¿öãÔ‡Ñç % disk.nvme0n1.out‚40950.3154296875 disk.dm-1.out‚ 276.66015625 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd2265.out‚0 network.sent‚ 1.7769138e+07 disk.nvme0n1.in‚ 1167.7109375 disk.sda3.out‚ 276.66015625 disk.drbd2266.out‚0 cpu‚25.360013191126313% disk./.usageGB‚247.87271118164062 disk.dm-1.in‚ 0.1953125 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚93.86880397796631* proc.memory.percent‚17.101335525512695 disk.rbd1.in‚0 disk.dm-0.in‚ 1167.7109375" disk.dm-0.out‚40950.3154296875 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2801.8828125 disk.sda1.out‚0 disk.drbd1716.in‚0) proc.memory.availableMB‚ 1004.53515625( disk./.usagePercent‚16.6349456264574 disk.drbd2266.in‚0 disk.sda2.in‚0 network.recv‚ 2.130614e+06 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚40950.3154296875 proc.cpu.threads‚17 disk.rbd1.out‚ 27.294921875 disk.sda.out‚ 276.66015625 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0>Æ´¹’Ÿ‚(:‡ ÎöãÔèÇÜ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚181.409' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œóôïs ‚(:è ÎöãÔ§”³  disk.drbd2080.out‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚277.3125 disk.rbd1.out‚ 27.5693359375 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2266.in‚0% disk./.usageGB‚247.90407943725586" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.in‚0 disk.dm-0.in‚ 1167.7109375 disk.dm-1.out‚277.3125 network.sent‚ 1.7775073e+07 disk.sda2.out‚0& disk.nvme0n1.out‚41021.68310546875 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2801.8828125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.1953125 cpu‚25.359167261152013 proc.cpu.threads‚17 disk.sda.out‚277.3125 network.recv‚ 2.134827e+06 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0# disk.dm-0.out‚41021.68310546875& proc.memory.availableMB‚ 1004.28125* proc.memory.percent‚17.101335525512695 disk.sda1.in‚0 disk.drbd2266.out‚0$ memory_percent‚93.87035369873047* disk./.usagePercent‚16.637050776411456 disk.rbd0.out‚0( disk.nvme0c0n1.out‚41021.68310546875Š Ðk2¡- _wandbruntime‚5809  _runtime‚5809ÈÛOȢ ' lr-AdamW/pg1‚0.00010606423371843812 ' lr-AdamW/pg2‚0.00010606423371843812 trainer/global_step‚7449 % _timestamp‚1.7884107060522609e+09 _runtime‚ 5809.03969163 _step‚610]cBꣂ trainer/global_step‚7449 % _timestamp‚1.7884107060522609e+09 _runtime‚ 5809.03969163 _step‚610 ' lr-AdamW/pg1‚0.00010606423371843812 ' lr-AdamW/pg2‚0.00010606423371843812â9{´2¤- _wandbruntime‚5809  _runtime‚5809`@|Cä'¥ÞO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108511464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140747935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433768784786944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99021663 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6551130448384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140767973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480186974342144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108511466 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480186784680960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚422392497146880 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6245410332160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108511466 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249279439 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244753969664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140747935 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚5809.040578623 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6551296169472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433768802789888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6244753969664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99022103 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63315812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6222414918383 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚611 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚440013538756608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108511464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚440014213122048 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440013538756608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚428614257627507 trainer/global_step‚7449 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473635654232576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221760480627 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35706289 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788410706052967e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433768784786944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚428614929104623 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚440014213122048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473635678172672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422392514186240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473635678172672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422392514186240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108511466 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249279439 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244753969664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140747935 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚5809.040578623 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6551296169472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433768802789888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6244753969664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99022103 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63315812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6222414918383 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚611 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚440013538756608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108511464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚440014213122048 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440013538756608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚428614257627507 trainer/global_step‚7449 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473635654232576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221760480627 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35706289 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788410706052967e+09ãbNâþ2§- _wandbruntime‚5809  _runtime‚5809æ¢kä¨éM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440070711223296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433825137825792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6245573397504 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108525482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108525482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480249291326464 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108525480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35710738 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552152064000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245573397504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291869 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚612 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99034288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63323980 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480249480974336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108525480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884107067469354e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140786453 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428670649670355 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6551986356224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222576915591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140766389 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433825155828736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_byt”‡¬ es.large_pool.freed‚422447401277440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433825137825792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6223231353555 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚7449 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246229773312 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5809.735309011 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066y… Ûô&©îM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108525480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35710738 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552152064000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245573397504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291869 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚612 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99034288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63323980 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480249480974336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108525480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884107067469354e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140786453 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428670649670355 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6551986356224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222576915591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140766389 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433825155828736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚422447401277440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433825137825792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6223231353555 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚7449 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246229773312 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5809.735309011 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440070711223296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433825137825792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6245573397504 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108525482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108525482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480249291326464äຮ€2ª- _wandbruntime‚5809  _runtime‚5809Ôl–gZ«Ô 2loss/SRqqt3/assignment_loss‚1.2762934305528935 /loss/FBt2/detection_loss‚2.6414024896091886 0loss/FRt2/assignment_loss‚3.0100686383247375 1loss/SRqqt1/assignment_loss‚1.722861932308385 1loss/FBt1/assignment_loss‚0.16495256929192692 1loss/FBt4/assignment_loss‚0.43073889033985324 /loss/FRt2/detection_loss‚2.3084418791532526 /loss/FRt4/detection_loss‚2.7462787220352576 epoch‚6 _runtime‚5809.736547367 0loss/FRt3/assignment_loss‚1.8471882410769194 /loss/FBt4/detection_loss‚2.3523813039064407 trainer/global_step‚7449 2loss/SRqqt4/assignment_loss‚1.5738116657095296 0loss/FBt2/assignment_loss‚0.6938545046270721 1loss/FBt3/assignment_loss‚0.19770536310940834 /loss/FRt1/detection_loss‚3.3258364722132683 .loss/FRt3/detection_loss‚1.821324013314157 2loss/SRqqt2/assignment_loss‚1.2286778054400986 1loss/SRqqt1/detection_loss‚2.6781586426677126 1loss/SRqqt4/detection_loss‚1.6550746450998954 % _timestamp‚1.7884107067491705e+09 .loss/FRt1/assignment_loss‚3.47188871105512 .loss/FBt1/detection_loss‚2.744949385523796 %loss/total_loss‚43.87684004269731 _step‚613 0loss/FRt4/assignment_loss‚2.7682596476454484 0loss/SRqqt3/detection_loss‚0.839894562556937 .loss/FBt3/detection_loss‚0.489658770800036 0loss/SRqqt2/detection_loss‚1.887137756335969"¡*_¬Ù .loss/FRt1/assignment_loss‚3.47188871105512 .loss/FBt1/detection_loss‚2.744949385523796 %loss/total_loss‚43.87684004269731 _step‚613 0loss/FRt4/assignment_loss‚2.7682596476454484 0loss/SRqqt3/detection_loss‚0.839894562556937 .loss/FBt3/detection_loss‚0.489658770800036 0loss/SRqqt2/detection_loss‚1.887137756335969 2loss/SRqqt3/assignment_loss‚1.2762934305528935 /loss/FBt2/detection_loss‚2.6414024896091886 0loss/FRt2/assignment_loss‚3.0100686383247375 1loss/SRqqt1/assignment_loss‚1.722861932308385 1loss/FBt1/assignment_loss‚0.16495256929192692 1loss/FBt4/assignment_loss‚0.43073889033985324 /loss/FRt2/detection_loss‚2.3084418791532526 /loss/FRt4/detection_loss‚2.7462787220352576 epoch‚6 _runtime‚5809.736547367 0loss/FRt3/assignment_loss‚1.8471882410769194 /loss/FBt4/detection_loss‚2.3523813039064407 trainer/global_step‚7449 2loss/SRqqt4/assignment_loss‚1.5738116657095296 0loss/FBt2/assignment_loss‚0.6938545046270721 1loss/FBt3/assignment_loss‚0.19770536310940834 /loss/FRt1/detection_loss‚3.3258364722132683 .loss/FRt3/detection_loss‚1.821324013314157 2loss/SRqqt2/assignment_loss‚1.2286778054400986 1loss/SRqqt1/detection_loss‚2.6781586426677126 1loss/SRqqt4/detection_loss‚1.6550746450998954 % _timestamp‚1.7884107067491705e+09å!ˆœ­‚(:‘ ÝöãÔËÔÆ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.193' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.83860000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èrDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6262595820035 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6594177589248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442872176504832 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442872176504832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442872850870272 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6262595820035 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6594177589248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442872176504832 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442872176504832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatËs‹6sMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ç>èü2·- _wandbruntime‚5846  _runtime‚58467&%‚ï&¸éM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250918721 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425192607805440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884107436436105e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436642807771136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250918721 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263412254999 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141712505 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99668757 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚436642807771136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7499 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109226282 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35936814 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431456020060439 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425192624844800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442930023350272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442929348971520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109226282 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431456691537763 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚436642789768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938787 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6286559203328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109226280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141712505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63732382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6595031683072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6264066692963 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938787 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63732380 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚476779865793536 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0úz_\ô&¹îM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚483374897476608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6594868072448 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141692441 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483374709925888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚476779841853440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6287215579136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6287215579136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35936377 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5846.631714521 _step‚616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6286559203328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109226280 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141692441 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436642789768192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚436642807771136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7499 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109226282 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35936814 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431456020060439 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425192624844800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442930023350272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442929348971520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109226282 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431456691537763 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚436642789768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938787 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6286559203328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109226280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141712505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63732382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6595031683072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6264066692963 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938787 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63732380 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚476779865793536 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250918721 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425192607805440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884107436436105e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436642807771136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250918721 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263412254999 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141712505 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99668757 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445730814253056 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚486437911380992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427882927242240 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445731488618496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6326725581312 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚415 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 5882.54480397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439404088671744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6304085597199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439404088671744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109913066 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100290483 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486437732205568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142620077 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142600039 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445730814253056 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36157869 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6327381943808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100290066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚479800751939072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36157454 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427882910202880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884107795570679e+09 _step‚619¨NbPë'ÈåO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439404088671744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109913066 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100290483 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486437732205568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142620077 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142600039 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445730814253056 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36157869 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6327381943808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100290066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚479800751939072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36157454 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427882910202880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884107795570679e+09 _step‚619 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6637004206592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445730814253056 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚486437911380992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427882927242240 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445731488618496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6326725581312 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚415 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 5882.54480397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439404088671744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6304085597199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2ëF¾u2É- _wandbruntime‚5883  _runtime‚5883ãŸÐñ&ÊëM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6328201384960 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434242061927847 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚439460441710592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64140780 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚479862378736640 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445787986719744 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚486500438209536 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109927082 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100302831 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36162051 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6327545009152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445788661098496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252545573 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486500259047424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚479862402676736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚620 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6304247594407 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109927080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109927082 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚439460459713536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6638035532800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6304902032371 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142618493 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445788661098496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565639 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427937814333440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚7549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚5883.245076065 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445787986719744 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚486500438209536 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109927082 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100302831 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36162051 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6327545009152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445788661098496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252545573 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486500259047424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚479862402676736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚620 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6304247594407 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109927080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109927082 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚439460459713536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6638035532800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6304902032371 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142618493 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445788661098496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565639 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427937814333440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚7549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚5883.245076065 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434242061927847 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944ìÚ[ª2Ì- _wandbruntime‚5883  _runtime‚5883=4gÁ\ÍÖ .loss/FRt2/detection_loss‚2.339854130657714 /loss/FRt4/detection_loss‚2.4329618748691346 /loss/FBt3/detection_loss‚0.6468352566169965 $ _timestamp‚1.788410780258772e+09 /loss/FBt2/assignment_loss‚0.189457799308002 1loss/SRqqt3/detection_loss‚0.7708022141953309 epoch‚6 trainer/global_step‚7549 _runtime‚5883.246126928 /loss/FRt3/assignment_loss‚1.202342100310744 1loss/FBt1/assignment_loss‚0.17772405010847642 /loss/FRt1/detection_loss‚2.8155961104801723 1loss/SRqqt2/detection_loss‚1.8131142796254625 _step‚621 0loss/FRt1/assignment_loss‚3.2074646813528878 /loss/FRt2/assignment_loss‚2.744379825708343 2loss/SRqqt3/assignment_loss‚0.9631487558285395 1loss/FBt3/assignment_loss‚0.11701232187318493 2loss/SRqqt1/assignment_loss‚1.7550296886609151 2loss/FBt4/assignment_loss‚0.001884199744987806 %loss/total_loss‚39.48591673662669 /loss/FRt3/detection_loss‚1.5113669326692296 0loss/SRqqt1/detection_loss‚2.434918081149077 /loss/FBt1/detection_loss‚2.8184323481151035 2loss/SRqqt4/assignment_loss‚1.3425861099579681 /loss/FBt4/detection_loss‚2.3099491894245148 2loss/SRqqt2/assignment_loss‚1.3783174255753265 1loss/SRqqt4/detection_loss‚1.9569208190465968 .loss/FBt2/detection_loss‚2.252512164413929 0loss/FRt4/assignment_loss‚2.3033063769340516& ÞõaÎÛ /loss/FBt1/detection_loss‚2.8184323481151035 2loss/SRqqt4/assignment_loss‚1.3425861099579681 /loss/FBt4/detection_loss‚2.3099491894245148 2loss/SRqqt2/assignment_loss‚1.3783174255753265 1loss/SRqqt4/detection_loss‚1.9569208190465968 .loss/FBt2/detection_loss‚2.252512164413929 0loss/FRt4/assignment_loss‚2.3033063769340516 .loss/FRt2/detection_loss‚2.339854130657714 /loss/FRt4/detection_loss‚2.4329618748691346 /loss/FBt3/detection_loss‚0.6468352566169965 $ _timestamp‚1.788410780258772e+09 /loss/FBt2/assignment_loss‚0.189457799308002 1loss/SRqqt3/detection_loss‚0.7708022141953309 epoch‚6 trainer/global_step‚7549 _runtime‚5883.246126928 /loss/FRt3/assignment_loss‚1.202342100310744 1loss/FBt1/assignment_loss‚0.17772405010847642 /loss/FRt1/detection_loss‚2.8155961104801723 1loss/SRqqt2/detection_loss‚1.8131142796254625 _step‚621 0loss/FRt1/assignment_loss‚3.2074646813528878 /loss/FRt2/assignment_loss‚2.744379825708343 2loss/SRqqt3/assignment_loss‚0.9631487558285395 1loss/FBt3/assignment_loss‚0.11701232187318493 2loss/SRqqt1/assignment_loss‚1.7550296886609151 2loss/FBt4/assignment_loss‚0.001884199744987806 %loss/total_loss‚39.48591673662669 /loss/FRt3/detection_loss‚1.5113669326692296 0loss/SRqqt1/detection_loss‚2.434918081149077íPÇ“PÏ‚(:’ ¨÷ãÔ⎚ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó PþuЂ(:ê ¨÷ãÔë·¨  disk.drbd2265.in‚0* proc.memory.percent‚17.101335525512695 network.recv‚ 2.163511e+06" disk.nvme0c0n1.in‚ 1167.7109375 disk.dm-1.out‚ 281.171875" proc.memory.rssMB‚ 2801.8828125 disk.drbd2266.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚41444.7275390625 cpu‚25.354519074533034 disk.rbd1.out‚ 28.1171875 disk.sda.out‚ 281.171875 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0% disk.nvme0n1.out‚41444.7275390625 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" disk.dm-0.out‚41444.7275390625 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚93.87180805206299) proc.memory.availableMB‚ 1004.04296875 disk.dm-0.in‚ 1167.7109375$ disk./.usageGB‚248.0073127746582 disk.sda3.out‚ 281.171875 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.sent‚ 1.8015515e+07* disk./.usagePercent‚16.643978852303068 disk.nvme0n1.in‚ 1167.7109375 disk.dm-1.in‚ 0.1953125’ >’Ñ‚(:‡ ·÷ãÔÔ€ñ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚185.297' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.0594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0, £uÒ‚(:ê ·÷ãÔä•Š  network.recv‚ 2.169471e+06 disk.drbd2080.out‚0 disk.rbd1.out‚ 28.1171875$ memory_percent‚93.86839866638184( proc.memory.availableMB‚ 1004.6015625" proc.memory.rssMB‚ 2801.8828125* proc.memory.percent‚17.101335525512695 disk.sda3.in‚ 0.1953125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 1167.7109375 proc.cpu.threads‚17 disk.rbd1.in‚0* disk./.usagePercent‚16.645800089901655% disk./.usageGB‚248.03445053100586 disk.sda.out‚ 281.890625 disk.sda3.out‚ 281.890625 disk.drbd2266.out‚0 disk.drbd1716.in‚0 network.sent‚ 1.8124282e+07 disk.rbd0.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1167.7109375' disk.nvme0c0n1.out‚41503.9970703125% disk.nvme0n1.out‚41503.9970703125 disk.dm-0.in‚ 1167.7109375 disk.dm-1.out‚ 281.890625 disk.drbd2080.in‚0 disk.drbd2266.in‚0" disk.dm-0.out‚41503.9970703125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 cpu‚25.353700637483904 disk.sda.in‚ 0.1953125 disk.sda2.in‚0 disk.drbd2265.in‚04]çC2Ó- _wandbruntime‚5919  _runtime‚5919§¼ä˜ÉÔà % _timestamp‚1.7884108163611705e+09 _runtime‚5919.348614769 _step‚622 ' lr-AdamW/pg1‚0.00010029734591032215 ' lr-AdamW/pg2‚0.00010029734591032215 trainer/global_step‚7599òÑ°|ÎÕÈ ' lr-AdamW/pg1‚0.00010029734591032215 ' lr-AdamW/pg2‚0.00010029734591032215 trainer/global_step‚7599 % _timestamp‚1.7884108163611705e+09 _runtime‚5919.348614769 _step‚622î¨ÄLœ2Ö- _wandbruntime‚5919  _runtime‚5919P¸oæ'×àO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139955 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442221758617088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64541014 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489563364632576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254159995 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6367711387136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448589452001280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442221758617088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436972383229731 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6344266498851 _runtime‚5919.349542169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143546129 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110613866 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6368367749632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448590126366720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448589452001280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442221740614144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110613866 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489563172874240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6344920936607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100924406 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430628133770240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6679907992064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143546129 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36383394 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6680075810304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110613864 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143526091 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100924857 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482883264882176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360‹ÃÚØåO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64541012 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 $ _timestamp‚1.788410816361893e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110613864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442221758617088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64541014 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489563364632576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254159995 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6367711387136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448589452001280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442221758617088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436972383229731 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6344266498851 _runtime‚5919.349542169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143546129 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110613866 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6368367749632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448590126366720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448589452001280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442221740614144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110613866 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489563172874240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6344920936607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100924406 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430628133770240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6679907992064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/Õ«@ allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143546129 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36383394 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6680075810304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110613864 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143526091 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100924857 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482883264882176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139955 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437028775272579 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6369187190784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6368530814976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36388250 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143564609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442278111655936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442278093652992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6369187190784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482944915619840 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192491 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7599 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110627882 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143544545 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚430683037900800 $ _timestamp‚1.788410817050633e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254172425 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6680928004096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100936983 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64549182 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36387803 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚448647298846720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6680760199168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368530814976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110627880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448646624467968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489625675819008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64549180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192491 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442278111655936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489625867564032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928‡Žkõ&ÛïM $ _timestamp‚1.788410817050633e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254172425 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6680928004096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100936983 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64549182 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36387803 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚448647298846720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6680760199168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368530814976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110627880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448646624467968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489625675819008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64549180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192491 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442278111655936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489625867564032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437028775272579 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6369187190784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6368530814976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36388250 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143564609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442278111655936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442278093652992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6369187190784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482944915619840 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192491 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7599 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110627882 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143544545 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚430683037900800ð¡u9)2Ü- _wandbruntime‚5920  _runtime‚5920°îÈ—XÝÒ trainer/global_step‚7599 _runtime‚ 5920.03997537 -loss/FRt1/detection_loss‚2.87891150812308 /loss/FRt3/detection_loss‚1.4362260245585017 $ _timestamp‚1.788410817052585e+09 0loss/FRt2/assignment_loss‚2.7476777479052545 0loss/FRt3/assignment_loss‚1.1375982942325729 /loss/FRt4/assignment_loss‚2.305203692479567 2loss/SRqqt1/assignment_loss‚2.2310114060387467 2loss/SRqqt3/assignment_loss‚1.0018560729093022 1loss/FBt1/assignment_loss‚0.01258413811434309 /loss/FBt4/assignment_loss‚0.602552464704786 1loss/SRqqt3/detection_loss‚1.1013406672928896 0loss/FRt1/assignment_loss‚3.1048120220502216 2loss/SRqqt2/assignment_loss‚1.4101433056705404 1loss/FBt2/assignment_loss‚0.21428562619257718 1loss/FBt3/assignment_loss‚0.12317842243646533 /loss/FBt2/detection_loss‚2.5581772550940514 _step‚625 1loss/SRqqt4/assignment_loss‚1.625357188734001 /loss/FRt2/detection_loss‚2.3743862032899465 /loss/FBt1/detection_loss‚2.7419892311096192 .loss/FBt4/detection_loss‚3.074835702776909 /loss/SRqqt2/detection_loss‚1.79993208483705 1loss/SRqqt4/detection_loss‚1.4825561184050708 /loss/FRt4/detection_loss‚2.4887046671726485 1loss/SRqqt1/detection_loss‚2.1954143661441226 &loss/total_loss‚41.101826304701575 epoch‚6 .loss/FBt3/detection_loss‚0.453092094429303Šš]Þ× _runtime‚ 5920.03997537 -loss/FRt1/detection_loss‚2.87891150812308 /loss/FRt3/detection_loss‚1.4362260245585017 $ _timestamp‚1.788410817052585e+09 0loss/FRt2/assignment_loss‚2.7476777479052545 0loss/FRt3/assignment_loss‚1.1375982942325729 /loss/FRt4/assignment_loss‚2.305203692479567 2loss/SRqqt1/assignment_loss‚2.2310114060387467 2loss/SRqqt3/assignment_loss‚1.0018560729093022 1loss/FBt1/assignment_loss‚0.01258413811434309 /loss/FBt4/assignment_loss‚0.602552464704786 1loss/SRqqt3/detection_loss‚1.1013406672928896 0loss/FRt1/assignment_loss‚3.1048120220502216 2loss/SRqqt2/assignment_loss‚1.4101433056705404 1loss/FBt2/assignment_loss‚0.21428562619257718 1loss/FBt3/assignment_loss‚0.12317842243646533 /loss/FBt2/detection_loss‚2.5581772550940514 _step‚625 1loss/SRqqt4/assignment_loss‚1.625357188734001 /loss/FRt2/detection_loss‚2.3743862032899465 /loss/FBt1/detection_loss‚2.7419892311096192 .loss/FBt4/detection_loss‚3.074835702776909 /loss/SRqqt2/detection_loss‚1.79993208483705 1loss/SRqqt4/detection_loss‚1.4825561184050708 /loss/FRt4/detection_loss‚2.4887046671726485 1loss/SRqqt1/detection_loss‚2.1954143661441226 &loss/total_loss‚41.101826304701575 epoch‚6 .loss/FBt3/detection_loss‚0.453092094429303 trainer/global_step‚7599ñîçß‚(:… Æ÷ãÔâòî gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚137.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç×&Â}à‚(:ò Æ÷ãÔâ†þ  disk.drbd1716.out‚0 disk.nvme0n1.in‚ 1167.7109375* proc.memory.percent‚17.101335525512695 disk.sda3.out‚ 282.33984375 disk.sda2.out‚0' disk.nvme0c0n1.out‚41538.2802734375 disk.rbd1.in‚0 disk.rbd1.out‚ 28.2431640625 disk.dm-0.in‚ 1167.7109375" disk.dm-0.out‚41538.2802734375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 282.33984375 disk.sda.in‚ 0.1953125 network.recv‚ 2.173651e+06 disk.drbd2080.in‚0$ memory_percent‚93.87013912200928" proc.memory.rssMB‚ 2801.8828125( disk./.usagePercent‚16.6477698118701 disk.drbd2266.out‚0% disk.nvme0n1.out‚41538.2802734375) proc.memory.availableMB‚ 1004.31640625 cpu‚25.352977708678136 proc.cpu.threads‚17% disk./.usageGB‚248.06380081176758 disk.sda3.in‚ 0.1953125 disk.sda.out‚ 282.33984375 network.sent‚ 1.8130116e+07 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.dm-1.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0§®|°’á‚(:‡ Õ÷ãÔŒ³™ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.918' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1836 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tβø{â‚(:ð Õ÷ãÔˆ¦¨ * disk./.usagePercent‚16.650813741452218 disk.drbd2265.in‚0 disk.sda3.out‚ 282.7890625$ memory_percent‚93.86858940124512 disk.rbd1.out‚ 28.2431640625 disk.nvme0n1.in‚ 1167.7109375" disk.dm-0.out‚41609.1103515625 disk.sda.in‚ 0.1953125 disk.sda.out‚ 282.7890625 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.sda2.in‚0' disk.nvme0c0n1.out‚41609.1103515625 cpu‚25.352221792752385 network.recv‚ 2.179935e+06 disk.drbd2266.out‚0( proc.memory.availableMB‚ 1004.5703125% disk./.usageGB‚248.10915756225586 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.101335525512695 disk.dm-1.out‚ 282.7890625 network.sent‚ 1.8239275e+07 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 1167.7109375 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚41609.1103515625 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2801.8828125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1167.7109375ã³ÔI‘ã‚(:† ä÷ãÔ‘£‚ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.411 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓD]»tä‚(:é ä÷ãÔ¬¡å proc.cpu.threads‚17 disk.dm-0.in‚ 1167.7109375 disk.dm-1.out‚ 283.71875 disk.sda.out‚ 283.71875 disk.drbd2266.in‚0 disk.sda2.out‚0 network.sent‚ 1.8244663e+07% disk.nvme0n1.out‚41681.7470703125 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125) proc.memory.availableMB‚ 1004.57421875 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2801.8828125* proc.memory.percent‚17.101335525512695) disk./.usagePercent‚16.65482358746102 disk.sda3.out‚ 283.71875" disk.nvme0c0n1.in‚ 1167.7109375 cpu‚25.351426564744607" disk.dm-0.out‚41681.7470703125 disk.drbd2080.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚41681.7470703125% disk./.usageGB‚248.16890716552734 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0# memory_percent‚93.8685655593872 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 28.3056640625 disk.drbd2265.out‚0 disk.drbd2080.out‚0 network.recv‚ 2.183634e+06 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0eaq2å- _wandbruntime‚5955  _runtime‚5955—´´ÇæÁ _step‚626 & lr-AdamW/pg1‚9.839003559347027e-05 & lr-AdamW/pg2‚9.839003559347027e-05 trainer/global_step‚7649 % _timestamp‚1.7884108530067513e+09 _runtime‚5955.994216577nxýÚÌçÆ & lr-AdamW/pg1‚9.839003559347027e-05 & lr-AdamW/pg2‚9.839003559347027e-05 trainer/global_step‚7649 % _timestamp‚1.7884108530067513e+09 _runtime‚5955.994216577 _step‚626òç•=v2è- _wandbruntime‚5955  _runtime‚5955ç<¹]éàO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451448764114944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101559520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚5955.995141773 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144472181 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144472181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6408697192960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111314664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439759096574255 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64949414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445039392556544 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451448089749504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451448764114944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101559959 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36610108 trainer/global_step‚7649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6723050536960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111314664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144452143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255786847 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492688690775552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433373323258880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ¦áë'êåO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255766807 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144452143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6722889010176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445039392556544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6409353555456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6385101838259 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451448764114944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101559520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚5955.995141773 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144472181 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144472181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6408697192960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111314664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439759096574255 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64949414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445039392556544 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451448089749504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451448764114944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101559959 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36610108 trainer/global_step‚7649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6723050536960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111314664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144452143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255786847 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492688690775552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433373323258880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101572635 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6409516620800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433428244428800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6410172996608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36615053 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445095745595392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6385918273223 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144470597 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255799277 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101572190 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492751389807104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255819343 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111328680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6723753947648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451505262216192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144490661 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445095763598336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819343 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144490661 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36614610 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6723913363968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445095763598336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚5956.681453587 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6409516620800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6410172996608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492751206450688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928=pAÌFíðM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101572635 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6409516620800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433428244428800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6410172996608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36615053 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445095745595392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6385918273223 K=DeviceStatsMonitor.on_trµ°ain_batch_end/active.small_pool.freed‚ 144470597 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255799277 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101572190 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492751389807104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255819343 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111328680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6723753947648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451505262216192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144490661 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445095763598336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819343 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144490661 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36614610 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6723913363968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445095763598336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚5956.681453587 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6409516620800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6410172996608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492751206450688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ôg¯Õ2î- _wandbruntime‚5956  _runtime‚5956ˆÐâ VïÐ /loss/SRqqt4/detection_loss‚1.58192940505928 .loss/FRt1/detection_loss‚2.799311183966123 .loss/FRt2/assignment_loss‚2.77154876840742 2loss/SRqqt1/assignment_loss‚1.8875900567058594 2loss/SRqqt2/assignment_loss‚1.3311591322666831 1loss/SRqqt3/detection_loss‚1.1573183720276967 -loss/FBt2/detection_loss‚2.01450252532959 _step‚629 1loss/FBt3/assignment_loss‚0.12092217573496374 trainer/global_step‚7649 _runtime‚5956.683017509 %loss/total_loss‚40.56742650923742 % _timestamp‚1.7884108536956146e+09 .loss/FRt1/assignment_loss‚3.90479166385455 2loss/SRqqt4/assignment_loss‚1.5068986154450426 /loss/FBt1/assignment_loss‚0.219949533996972 /loss/FRt3/detection_loss‚1.6354041560713588 1loss/SRqqt2/detection_loss‚1.7790486913706576 .loss/FBt1/detection_loss‚3.016963948806127 0loss/FRt4/assignment_loss‚2.6659559708721234 /loss/FRt2/detection_loss‚2.1793518053437406 .loss/FRt4/detection_loss‚2.048869933300423 /loss/FBt4/detection_loss‚2.1436474160714583 1loss/FBt4/assignment_loss‚0.08194479933643545 /loss/FBt3/detection_loss‚0.5229690202545109 epoch‚6 2loss/SRqqt3/assignment_loss‚0.8752780486200307 1loss/SRqqt1/detection_loss‚2.5352900662729816 0loss/FRt3/assignment_loss‚1.5840448860464424 1loss/FBt2/assignment_loss‚0.20273633407694952ë¶[d[ðÕ 2loss/SRqqt3/assignment_loss‚0.8752780486200307 1loss/SRqqt1/detection_loss‚2.5352900662729816 0loss/FRt3/assignment_loss‚1.5840448860464424 1loss/FBt2/assignment_loss‚0.20273633407694952 /loss/SRqqt4/detection_loss‚1.58192940505928 .loss/FRt1/detection_loss‚2.799311183966123 .loss/FRt2/assignment_loss‚2.77154876840742 2loss/SRqqt1/assignment_loss‚1.8875900567058594 2loss/SRqqt2/assignment_loss‚1.3311591322666831 1loss/SRqqt3/detection_loss‚1.1573183720276967 -loss/FBt2/detection_loss‚2.01450252532959 _step‚629 1loss/FBt3/assignment_loss‚0.12092217573496374 trainer/global_step‚7649 _runtime‚5956.683017509 %loss/total_loss‚40.56742650923742 % _timestamp‚1.7884108536956146e+09 .loss/FRt1/assignment_loss‚3.90479166385455 2loss/SRqqt4/assignment_loss‚1.5068986154450426 /loss/FBt1/assignment_loss‚0.219949533996972 /loss/FRt3/detection_loss‚1.6354041560713588 1loss/SRqqt2/detection_loss‚1.7790486913706576 .loss/FBt1/detection_loss‚3.016963948806127 0loss/FRt4/assignment_loss‚2.6659559708721234 /loss/FRt2/detection_loss‚2.1793518053437406 .loss/FRt4/detection_loss‚2.048869933300423 /loss/FBt4/detection_loss‚2.1436474160714583 1loss/FBt4/assignment_loss‚0.08194479933643545 /loss/FBt3/detection_loss‚0.5229690202545109 epoch‚6õˆë–ñ‚(:‚ ó÷ãÔ”õŸ gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏpÝ%|ò‚(:ñ ó÷ãÔ¼Ž©  cpu‚25.35066771081197" proc.memory.rssMB‚ 2801.8828125 network.recv‚ 2.187803e+06 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2265.out‚0 disk.sda.out‚ 284.19921875) proc.memory.availableMB‚ 1004.05859375) disk./.usagePercent‚16.66087355751019 disk.rbd1.in‚0" disk.dm-0.out‚41741.6142578125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 284.19921875 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 284.19921875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0$ disk./.usageGB‚248.2590560913086 disk.rbd1.out‚ 28.3056640625 disk.dm-0.in‚ 1167.7109375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1167.7109375$ memory_percent‚93.87171268463135 proc.cpu.threads‚17 disk.sda.in‚ 0.1953125' disk.nvme0c0n1.out‚41741.6142578125* proc.memory.percent‚17.101335525512695 network.sent‚ 1.8250625e+07% disk.nvme0n1.out‚41741.6142578125 disk.sda3.in‚ 0.1953125 disk.sda2.in‚0(&<ó‚(:… ‚øãÔЄú gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.61' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.122 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªdBzô‚(:ï ‚øãÔ×úæ# disk.dm-0.out‚41800.49365234375 network.sent‚ 1.8359264e+07 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1167.7109375$ memory_percent‚93.87173652648926 disk.dm-1.out‚ 284.8828125 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2801.8828125 disk.sda.out‚ 284.8828125( proc.memory.availableMB‚ 1004.0546875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚ 1167.7109375 disk.sda.in‚ 0.1953125* proc.memory.percent‚17.101335525512695 disk.sda2.out‚0 disk.nvme0n1.in‚ 1167.7109375& disk.nvme0n1.out‚41800.49365234375 disk.drbd2265.out‚0 cpu‚25.34988022645989) disk./.usagePercent‚16.66364611910601 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 28.494140625 disk.sda1.out‚0$ disk./.usageGB‚248.3003692626953 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚41800.49365234375 network.recv‚ 2.194067e+06 disk.sda3.out‚ 284.8828125 disk.drbd2080.in‚0 disk.dm-1.in‚ 0.1953125éÌPt2õ- _wandbruntime‚5992  _runtime‚5992?h¹ÇöÁ trainer/global_step‚7699 % _timestamp‚1.7884108896296122e+09 _runtime‚5992.617056269 _step‚630 & lr-AdamW/pg1‚9.649088188752804e-05 & lr-AdamW/pg2‚9.649088188752804e-05ÓðiäÌ÷Æ & lr-AdamW/pg1‚9.649088188752804e-05 & lr-AdamW/pg2‚9.649088188752804e-05 trainer/global_step‚7699 % _timestamp‚1.7884108896296122e+09 _runtime‚5992.617056269 _step‚630ök8 2ø- _wandbruntime‚5992  _runtime‚5992Q°‹Ïç'ùáO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚436118529786880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145378195 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚442544466964547 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145398233 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495814331019264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚7699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6449682998784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495814132969472 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚436118546826240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257413699 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102195197 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447857044498944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6449682998784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112015466 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚631 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6426591615423 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36837383 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145398233 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454306727497728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚5992.617950747 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6765968430592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6450339361280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454306727497728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6425937177667 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447857044498944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65357812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102194742 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489048338648576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454307401863168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145378195 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36836930 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112015466 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65357814 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454307401863168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447857062501888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚5992.617950747 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6765968430592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6450339361280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454306727497728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6425937177667 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447857044498944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65357812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102194742 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489048338648576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454307401863168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145378195 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36836930 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112015466 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65357814 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454307401863168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447857062501888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚442544466964547 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145398233 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495814331019264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚7699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6449682998784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495814132969472 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚436118546826240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 V;À˜´ÒHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257413699 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102195197 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0÷ÏDåR2û- _wandbruntime‚5993  _runtime‚5993¾DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6450502426624 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102207352 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚7699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495876829123584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6427408050595 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112029482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447913397537792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257426129 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65365982 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112029480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145396649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36841372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454363899964416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257426129 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257446195 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489109989386240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6766647992320 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447913397537792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚489110013326336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436173433917440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145416713 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446195 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112029480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436173450956800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚632 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454364574343168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ö^¯Úö&ýðM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6766647992320 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447913397537792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚489110013326336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436173433917440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145416713 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446195 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112029480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436173450956800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚632 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454364574343168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6450502426624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495876637378560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145416713 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102207790 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447913415540736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6451158802432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145396649 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6450502426624 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102207352 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚7699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495876829123584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6427408050595 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112029482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447913397537792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257426129 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65365982 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112029480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145396649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36841372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454363899964416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257426129 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257446195 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489109989386240øSÝN2þ- _wandbruntime‚5993  _runtime‚5993é™2TÿÎ .loss/FRt1/detection_loss‚2.718992647371794 2loss/SRqqt3/assignment_loss‚0.7922246889578864 1loss/SRqqt4/detection_loss‚1.6943660616874694 .loss/FBt1/detection_loss‚2.423253229686192 .loss/FBt2/detection_loss‚2.564824334212712 .loss/FRt4/assignment_loss‚2.77599186450243 2loss/FBt4/assignment_loss‚0.021220132398108642 _step‚633 1loss/FBt3/assignment_loss‚0.08263116646217258 /loss/FRt2/detection_loss‚2.0430831953655715 0loss/FRt1/assignment_loss‚3.3794342012781846 /loss/FRt3/assignment_loss‚1.163250276675591 1loss/SRqqt4/assignment_loss‚1.221666490320455 1loss/FBt1/assignment_loss‚0.32774316657534136 $ _timestamp‚1.788410890321133e+09 .loss/FRt4/detection_loss‚2.181011841942867 %loss/total_loss‚40.79411402772551 epoch‚6 /loss/FRt2/assignment_loss‚3.342055090095686 0loss/FBt2/assignment_loss‚0.3062287984149797 /loss/FBt3/detection_loss‚0.6112012872245254 1loss/SRqqt1/assignment_loss‚1.424932329977552 1loss/SRqqt2/detection_loss‚1.9011224084778835 _runtime‚5993.308461861 1loss/SRqqt2/assignment_loss‚1.100526334162344 1loss/SRqqt3/detection_loss‚1.1060887037269902 trainer/global_step‚7699 -loss/FRt3/detection_loss‚1.96356843970716 1loss/SRqqt1/detection_loss‚2.3849766472975413 .loss/FBt4/detection_loss‚3.263720691204071 ¬ÒÓY€Ó 1loss/FBt3/assignment_loss‚0.08263116646217258 /loss/FRt2/detection_loss‚2.0430831953655715 0loss/FRt1/assignment_loss‚3.3794342012781846 /loss/FRt3/assignment_loss‚1.163250276675591 1loss/SRqqt4/assignment_loss‚1.221666490320455 1loss/FBt1/assignment_loss‚0.32774316657534136 $ _timestamp‚1.788410890321133e+09 .loss/FRt4/detection_loss‚2.181011841942867 %loss/total_loss‚40.79411402772551 epoch‚6 /loss/FRt2/assignment_loss‚3.342055090095686 0loss/FBt2/assignment_loss‚0.3062287984149797 /loss/FBt3/detection_loss‚0.6112012872245254 1loss/SRqqt1/assignment_loss‚1.424932329977552 1loss/SRqqt2/detection_loss‚1.9011224084778835 _runtime‚5993.308461861 1loss/SRqqt2/assignment_loss‚1.100526334162344 1loss/SRqqt3/detection_loss‚1.1060887037269902 trainer/global_step‚7699 -loss/FRt3/detection_loss‚1.96356843970716 1loss/SRqqt1/detection_loss‚2.3849766472975413 .loss/FBt4/detection_loss‚3.263720691204071 .loss/FRt1/detection_loss‚2.718992647371794 2loss/SRqqt3/assignment_loss‚0.7922246889578864 1loss/SRqqt4/detection_loss‚1.6943660616874694 .loss/FBt1/detection_loss‚2.423253229686192 .loss/FBt2/detection_loss‚2.564824334212712 .loss/FRt4/assignment_loss‚2.77599186450243 2loss/FBt4/assignment_loss‚0.021220132398108642 _step‚633ù€ÚÿÖ’‚(:‡ ‘øãÔ’åÜ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚175.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y>x~‚‚(:ó ‘øãÔ¢ó¿ ) proc.memory.availableMB‚ 1080.80859375 disk.drbd1716.in‚0& disk.nvme0n1.out‚41848.01318359375 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 28.580078125 cpu‚25.349168990255222( disk.nvme0c0n1.out‚41848.01318359375 disk.rbd0.in‚0 disk.dm-0.in‚ 1167.7109375 disk.dm-1.out‚ 285.16015625 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2720.984375) proc.memory.percent‚16.60757064819336% disk./.usageGB‚248.35358428955078 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0# memory_percent‚93.4032678604126 network.sent‚ 1.8365558e+07 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 1167.7109375# disk.dm-0.out‚41848.01318359375 disk.sda.in‚ 0.1953125* disk./.usagePercent‚16.667217424208662 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 285.16015625 network.recv‚ 2.198518e+06 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 285.16015625 disk.rbd0.out‚09µnÏ’ƒ‚(:‡  øãÔ«†¨ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ûap„‚(:å  øãÔš§  cpu‚25.3483879981536 proc.cpu.threads‚17 network.recv‚ 2.203971e+06 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚285.8125" disk.nvme0c0n1.in‚ 1167.7109375) proc.memory.availableMB‚ 1080.80078125% disk./.usageGB‚248.39459609985352 disk.drbd1716.out‚0 disk.rbd1.out‚ 28.642578125& disk.nvme0n1.out‚41918.74169921875 disk.dm-0.in‚ 1167.7109375# disk.dm-0.out‚41918.74169921875 disk.drbd2265.in‚0 disk.sda3.out‚285.8125( disk.nvme0c0n1.out‚41918.74169921875 disk.drbd1716.in‚0) proc.memory.percent‚16.60757064819336 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚93.40331554412842! proc.memory.rssMB‚ 2720.984375 network.sent‚ 1.8473551e+07 disk.drbd2265.out‚0 disk.sda.out‚285.8125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0) disk./.usagePercent‚16.66996976120928 disk.sda3.in‚ 0.1953125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1167.7109375öÆné2…- _wandbruntime‚6029  _runtime‚6029!±/ÒdžÁ _runtime‚6029.187598025 _step‚634 & lr-AdamW/pg1‚9.460028057328118e-05 & lr-AdamW/pg2‚9.460028057328118e-05 trainer/global_step‚7749 % _timestamp‚1.7884109262001832e+09Õ8#ÖÌ‡Æ _step‚634 & lr-AdamW/pg1‚9.460028057328118e-05 & lr-AdamW/pg2‚9.460028057328118e-05 trainer/global_step‚7749 % _timestamp‚1.7884109262001832e+09 _runtime‚6029.187598025út2"—2ˆ- _wandbruntime‚6029  _runtime‚6029’¹—½‰àO trainer/global_step‚7749 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457166039611392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112716266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498939734279168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6808662794752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438863753354240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6491325167104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884109262009315e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37064063 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445331180309071 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactivá4’o)e_split.large_pool.freed‚65766212 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚635 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457165365245952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492130899471872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146324285 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450674714444288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6808834807296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457166039611392 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146304247 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445330508831955 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457165365245952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146324285 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102830734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112716264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚450674696441344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450674696441344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112716264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6466772517075 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6491325167104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844jÒÍ®ë'ŠåO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457166039611392 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146304247 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445330508831955 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457165365245952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146324285 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102830734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112716264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚450674696441344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450674696441344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112716264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6466772517075 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6491325167104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 trainer/global_step‚7749 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457166039611392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112716266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498939734279168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6808662794752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438863753354240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6491325167104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884109262009315e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37064063 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445331180309071 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65766212 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚635 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457165365245952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492130899471872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146324285 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450674714444288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6808834807296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ûE»Ý2‹- _wandbruntime‚6029  _runtime‚6029V çñ&ŒëM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146342765 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438918640445440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6492144608256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491488232448 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚636 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112730282 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146342765 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457222537712640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499002048092160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6468243390003 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65774382 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884109269008377e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37068608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457223212091392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146322701 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6467588952039 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445386229397479 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259073047 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450731049480192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6492144608256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102842988 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499002237740032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457223212091392 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚6029.889097716 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65774380 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450731067483136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445386900874803 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492192550209536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112730280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424«@u[¦ðM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457223212091392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146322701 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6467588952039 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445386229397479 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259073047 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450731049480192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6492144608256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102842988 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499002237740032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457223212091392 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚6029.889097716 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65774380 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450731067483136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445386900874803 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492192550209536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112730280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146342765 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438918640445440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6492144608256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491488232448 E=Devic?²’PeStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚636 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112730282 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146342765 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457222537712640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499002048092160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6468243390003 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65774382 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884109269008377e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37068608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872üœÜ2Ž- _wandbruntime‚6029  _runtime‚6029û¸ç·\Ö 2loss/SRqqt4/assignment_loss‚1.4200326871871949 0loss/SRqqt2/detection_loss‚2.060622498111905 1loss/SRqqt3/detection_loss‚0.8247207066811183 trainer/global_step‚7749 _runtime‚6029.890289325 .loss/FRt1/assignment_loss‚3.43951194132528 2loss/SRqqt2/assignment_loss‚1.5627145109311589 0loss/FRt3/assignment_loss‚1.2921363105376562 0loss/FRt4/assignment_loss‚2.8671354398131372 0loss/FBt4/assignment_loss‚0.1987363739986904 /loss/FRt1/detection_loss‚2.8606698955259016 .loss/FRt4/detection_loss‚2.723551942408085 _step‚637 2loss/SRqqt3/assignment_loss‚1.0245313724119436 1loss/FBt2/assignment_loss‚0.32027905681995134 1loss/SRqqt1/detection_loss‚2.4388613893139746 .loss/FBt4/detection_loss‚2.762691247463226 /loss/FRt2/detection_loss‚2.4536101548455873 /loss/FRt3/detection_loss‚1.3463408800462882 /loss/FBt1/detection_loss‚2.4205520927906035 1loss/FBt3/assignment_loss‚0.08038824700544099 .loss/FBt3/detection_loss‚0.732409612218171 &loss/total_loss‚41.914198928421236 epoch‚6 % _timestamp‚1.7884109269028816e+09 1loss/SRqqt4/detection_loss‚1.9346051958203316 /loss/FBt2/detection_loss‚2.0118784035245576 2loss/SRqqt1/assignment_loss‚1.6920311950868177 0loss/FBt1/assignment_loss‚0.2614291369215721 0loss/FRt2/assignment_loss‚3.1847586376326427ÓLðaÛ /loss/FRt1/detection_loss‚2.8606698955259016 .loss/FRt4/detection_loss‚2.723551942408085 _step‚637 2loss/SRqqt3/assignment_loss‚1.0245313724119436 1loss/FBt2/assignment_loss‚0.32027905681995134 1loss/SRqqt1/detection_loss‚2.4388613893139746 .loss/FBt4/detection_loss‚2.762691247463226 /loss/FRt2/detection_loss‚2.4536101548455873 /loss/FRt3/detection_loss‚1.3463408800462882 /loss/FBt1/detection_loss‚2.4205520927906035 1loss/FBt3/assignment_loss‚0.08038824700544099 .loss/FBt3/detection_loss‚0.732409612218171 &loss/total_loss‚41.914198928421236 epoch‚6 % _timestamp‚1.7884109269028816e+09 1loss/SRqqt4/detection_loss‚1.9346051958203316 /loss/FBt2/detection_loss‚2.0118784035245576 2loss/SRqqt1/assignment_loss‚1.6920311950868177 0loss/FBt1/assignment_loss‚0.2614291369215721 0loss/FRt2/assignment_loss‚3.1847586376326427 2loss/SRqqt4/assignment_loss‚1.4200326871871949 0loss/SRqqt2/detection_loss‚2.060622498111905 1loss/SRqqt3/detection_loss‚0.8247207066811183 trainer/global_step‚7749 _runtime‚6029.890289325 .loss/FRt1/assignment_loss‚3.43951194132528 2loss/SRqqt2/assignment_loss‚1.5627145109311589 0loss/FRt3/assignment_loss‚1.2921363105376562 0loss/FRt4/assignment_loss‚2.8671354398131372 0loss/FBt4/assignment_loss‚0.1987363739986904ý³êBt‘‚(:… ¯øãÔÀ¥ò gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡B}’‚(:ò ¯øãÔˆ²Ž  disk.sda2.out‚0 disk.rbd1.out‚ 28.642578125# disk.dm-0.out‚41965.09716796875 disk.drbd2265.in‚0 disk.sda3.out‚ 286.58984375 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125# memory_percent‚93.4037446975708 proc.cpu.threads‚17* disk./.usagePercent‚16.672170145966074% disk./.usageGB‚248.42738342285156 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2080.out‚0 disk.rbd0.out‚0 network.sent‚ 1.8479581e+07 disk.drbd1716.in‚0 disk.dm-0.in‚ 1167.7109375) proc.memory.availableMB‚ 1080.73046875 cpu‚25.34764027030079 disk.drbd2266.in‚0 network.recv‚ 2.208495e+06 disk.sda2.in‚0( disk.nvme0c0n1.out‚41965.09716796875& disk.nvme0n1.out‚41965.09716796875 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 286.58984375! proc.memory.rssMB‚ 2720.984375 disk.drbd2266.out‚0 disk.sda1.out‚0) proc.memory.percent‚16.60757064819336" disk.nvme0c0n1.in‚ 1167.7109375 disk.sda.out‚ 286.58984375 disk.rbd0.in‚0¢<;Î’“‚(:‡ ¾øãÔåí gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.124' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B33=€”‚(:õ ¾øãÔúú¢  disk.rbd0.out‚0# disk.dm-0.out‚42020.16162109375& disk.nvme0n1.out‚42020.16162109375% disk./.usageGB‚248.45494079589844 disk.dm-1.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 1167.7109375 network.sent‚ 1.8587104e+07* disk./.usagePercent‚16.674019544393424 disk.drbd1716.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0( proc.memory.availableMB‚ 1081.2578125" proc.memory.rssMB‚ 2724.7265625) proc.memory.percent‚16.63041114807129 disk.drbd2080.in‚0$ memory_percent‚93.40052604675293 cpu‚25.346853373079323 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1167.7109375 disk.dm-1.out‚ 287.33984375 network.recv‚ 2.213659e+06 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.sda1.out‚0( disk.nvme0c0n1.out‚42020.16162109375 disk.rbd1.out‚ 28.7685546875 disk.sda.out‚ 287.33984375 disk.nvme0n1.in‚ 1167.7109375 disk.sda3.out‚ 287.33984375 disk.drbd2080.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17[è¤ü~•‚(:ó ÍøãÔ¢§â disk.dm-1.in‚ 0.1953125 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 288.0078125 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0% disk./.usageGB‚248.47237396240234 disk.dm-0.in‚ 1167.7109375 disk.sda.out‚ 288.0078125 disk.drbd2080.out‚0 disk.rbd0.out‚0$ memory_percent‚93.40050220489502* disk./.usagePercent‚16.675189498824906 disk.drbd1716.out‚0 disk.rbd1.out‚ 28.7685546875& disk.nvme0n1.out‚42071.72802734375 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚42071.72802734375" proc.memory.rssMB‚ 2724.7265625) proc.memory.percent‚16.63041114807129 network.sent‚ 1.8593449e+07 network.recv‚ 2.218169e+06 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 288.0078125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1167.7109375) proc.memory.availableMB‚ 1081.26171875 cpu‚25.346102280025615 disk.nvme0n1.in‚ 1167.7109375# disk.dm-0.out‚42071.72802734375 disk.drbd2266.out‚0Úïþ¡’–‚(:‡ ÍøãÔ¿þñ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãî¥.2—- _wandbruntime‚6066  _runtime‚6066B˜HDǘÁ & lr-AdamW/pg1‚9.271862564920993e-05 & lr-AdamW/pg2‚9.271862564920993e-05 trainer/global_step‚7799 % _timestamp‚1.7884109630767086e+09 _runtime‚6066.064145827 _step‚638'A¡ÉÌ™Æ % _timestamp‚1.7884109630767086e+09 _runtime‚6066.064145827 _step‚638 & lr-AdamW/pg1‚9.271862564920993e-05 & lr-AdamW/pg2‚9.271862564920993e-05 trainer/global_step‚7799þaéP2š- _wandbruntime‚6066  _runtime‚6066Rî¦ç'›áO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460024677359616 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147250337 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66174614 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103465198 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6507607856483 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453492348383744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚7799 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6532310972928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460024002994176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚502065065014784 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113417066 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6532310972928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441608959882240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460024677359616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚423 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚421 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502065239995904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6508262294239 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453492366386688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚6066.065108453 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113417064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147250337 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6851803640832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147230299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495213436355072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453492348383744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37290586 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260667403 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460024002994176 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚453492366386688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 174981120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448117222176479 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441608942842880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495213412414976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6851652599808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113417066 _step‚639 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6531654610432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944Ú›ðì'œæO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103465198 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6507607856483 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453492348383744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚7799 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6532310972928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460024002994176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚502065065014784 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113417066 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6532310972928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441608959882240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460024677359616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚423 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚421 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502065239995904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6508262294239 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453492366386688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚6066.065108453 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113417064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147250337 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6851803640832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147230299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495213436355072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453492348383744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37290586 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260667403 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460024002994176 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚453492366386688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 174981120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448117222176479 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441608942842880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495213412414976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6851652599808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113417066 _step‚639 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6531654610432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460024677359616 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147250337 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66174614 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ÿšÛƒJ2- _wandbruntime‚6066  _runtime‚6066ÃÇm žêM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453548719425536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚448172271264887 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441663846973440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚6066.756081158 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66182780 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6508424291447 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260679833 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113431080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147248753 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495275063152640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7799 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6852684269056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 $ _timestamp‚1.788410963767966e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453548701422592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6533130414080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448172942742211 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113431080 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460081849839616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502127771361792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6852531144192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147248753 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460081849839616 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260699899 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103478276 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441663864012800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103477855 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502127594296832 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699899 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147268817 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6532474038272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚419 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113431082 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260679833 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66182782 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453548701422592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147268817 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495275087092736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6532474038272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453548719425536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37295075 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6509078729411 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113431082 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6508424291447 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260679833 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113431080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147248753 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495275063152640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7799 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6852684269056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 $ _timestamp‚1.788410963767966e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453548701422592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6533130414080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448172942742211 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113431080 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460081849839616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502127771361792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6852531144192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147248753 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460081849839616 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260699899 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103478276 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441663864012800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103477855 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502127594296832 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699899 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147268817 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6532474038272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚419 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113431082 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260679833 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66182782 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453548701422592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147268817 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495275087092736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6532474038272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453548719425536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37295075 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6509078729411 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113431082 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37518093 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6894583042048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444354166410240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462883315107840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114117866 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6572640416256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37518517 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148176389 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66583012 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6573296778752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262294255 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148156351 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884110001301422e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262274215 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114117866 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462882640742400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6549097633647 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114117864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456310018329088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548443195891 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114117864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66583014 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6573296778752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462882640742400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚6103.117833151 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 trainer/global_step‚7849 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456310018329088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6572640416256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104101105 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚450903264043887 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884110001301422e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262274215 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114117866 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462882640742400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6549097633647 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114117864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456310018329088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548443195891 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114117864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66583014 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6573296778752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462882640742400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚6103.117833151 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 trainer/global_step‚7849 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456310018329088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6572640416256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104101105 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚450903264043887 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37518093 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6894583042048ƒ{1_Õ2­- _wandbruntime‚6103  _runtime‚6103Ø×ì‰ð&®êM P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148194869 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326751 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456366353364992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444409053501440 _runtime‚6103.814025369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6573459844096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114131882 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498357623975936 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450958984609619 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456366371367936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104113740 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262306685 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114131880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462940487587840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104114186 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6574116219904 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456366353364992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462939813209088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6574116219904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66591180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456366371367936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262306685 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6573459844096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498357600035840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6549259630855 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262326751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚644 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¢!A€õ&¯ïM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262306685 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114131880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462940487587840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104114186 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6574116219904 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456366353364992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462939813209088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6574116219904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66591180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456366371367936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262306685 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6573459844096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498357600035840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6549259630855 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262326751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚644 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148194869 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326751 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456366353364992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444409053501440 _runtime‚6103.814025369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6573459844096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114131882 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498357623975936 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450958984609619 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456366371367936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104113740 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560„„¬32°- _wandbruntime‚6103  _runtime‚6103>rf‚±Ù _runtime‚6103.815109625 0loss/FBt1/assignment_loss‚0.3575083809714256 1loss/SRqqt4/detection_loss‚1.7888197721197054 /loss/FBt4/detection_loss‚2.7125030010938644 trainer/global_step‚7849 % _timestamp‚1.7884110008277402e+09 .loss/FRt1/detection_loss‚3.623241037130356 0loss/FRt1/assignment_loss‚3.6134402990341186 /loss/FRt2/assignment_loss‚2.667148865186251 2loss/SRqqt2/assignment_loss‚1.6643491559440182 1loss/FBt4/assignment_loss‚0.06696540805205586 /loss/FRt2/detection_loss‚2.1747562702117738 /loss/FRt3/detection_loss‚1.4511554841444176 1loss/SRqqt1/detection_loss‚2.1330255054765277 1loss/SRqqt3/detection_loss‚0.7300116321918639 /loss/FBt3/detection_loss‚0.4087123905724667 epoch‚6 0loss/FRt3/assignment_loss‚1.2876946453778249 2loss/SRqqt4/assignment_loss‚1.2477310250441616 /loss/FBt2/detection_loss‚ÇqåvÝ2.2505822512838574 _step‚645 0loss/FRt4/assignment_loss‚1.9495648881222338 1loss/FBt2/assignment_loss‚0.15438567510702544 .loss/FRt4/detection_loss‚2.265188675294531 1loss/SRqqt2/detection_loss‚2.0944437867119197 %loss/total_loss‚39.40753870482209 2loss/SRqqt1/assignment_loss‚1.5544387381523848 2loss/SRqqt3/assignment_loss‚0.8556152511197566 1loss/FBt3/assignment_loss‚0.07968139128090418 /loss/FBt1/detection_loss‚2.2765751751986416u 8d²Þ 2loss/SRqqt3/assignment_loss‚0.8556152511197566 1loss/FBt3/assignment_loss‚0.07968139128090418 /loss/FBt1/detection_loss‚2.2765751751986416 _runtime‚6103.815109625 0loss/FBt1/assignment_loss‚0.3575083809714256 1loss/SRqqt4/detection_loss‚1.7888197721197054 /loss/FBt4/detection_loss‚2.7125030010938644 trainer/global_step‚7849 % _timestamp‚1.7884110008277402e+09 .loss/FRt1/detection_loss‚3.623241037130356 0loss/FRt1/assignment_loss‚3.6134402990341186 /loss/FRt2/assignment_loss‚2.667148865186251 2loss/SRqqt2/assignment_loss‚1.6643491559440182 1loss/FBt4/assignment_loss‚0.06696540805205586 /loss/FRt2/detection_loss‚2.1747562702117738 /loss/FRt3/detection_loss‚1.4511554841444176 1loss/SRqqt1/detection_loss‚2.1330255054765277 1loss/SRqqt3/detection_loss‚0.7300116321918639 /loss/FBt3/detection_loss‚0.4087123905724667 epoch‚6 0loss/FRt3/assignment_loss‚1.2876946453778249 2loss/SRqqt4/assignment_loss‚1.2477310250441616 /loss/FBt2/detection_loss‚2.2505822512838574 _step‚645 0loss/FRt4/assignment_loss‚1.9495648881222338 1loss/FBt2/assignment_loss‚0.15438567510702544 .loss/FRt4/detection_loss‚2.265188675294531 1loss/SRqqt2/detection_loss‚2.0944437867119197 %loss/total_loss‚39.40753870482209 2loss/SRqqt1/assignment_loss‚1.5544387381523848…Ã6&›³‚(: úøãÔáØò gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.8' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.760000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í¿¯îo´‚(:ä úøãÔÖê * disk./.usagePercent‚16.588134648821562 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚42250.75146484375 disk.sda.out‚290.1875& proc.memory.availableMB‚ 1081.21875" disk.nvme0c0n1.in‚ 1167.7109375 disk.rbd1.in‚0 disk.dm-1.out‚290.1875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 1167.7109375 disk.dm-0.in‚ 1167.7109375 disk.sda.in‚ 0.1953125 network.sent‚ 1.8713374e+07 disk.rbd1.out‚ 28.95703125 disk.drbd2265.out‚0 disk.dm-1.in‚ 0.1953125 network.recv‚ 2.232213e+06$ disk./.usageGB‚247.1751937866211 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚290.1875 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0$ memory_percent‚93.40076446533203) proc.memory.percent‚16.63041114807129 cpu‚25.343878761952936 disk.drbd1716.out‚0& disk.nvme0n1.out‚42250.75146484375# disk.dm-0.out‚42250.75146484375" proc.memory.rssMB‚ 2724.7265625@N Ôµ‚(:’ ‰ùãÔŸ»² gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚187.905' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.580999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…ÛpÖ}¶‚(:ò ‰ùãÔÔŽ¦  disk.dm-1.in‚ 0.1953125( proc.memory.availableMB‚ 1081.1640625 disk.sda2.in‚0& disk.nvme0n1.out‚42302.38427734375 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0$ memory_percent‚93.40109825134277 cpu‚25.34315734695931 disk.dm-0.in‚ 1167.7109375 disk.rbd1.out‚ 28.95703125% disk./.usageGB‚247.18977737426758 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 1167.7109375) proc.memory.percent‚16.63041114807129 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0 network.recv‚ 2.238909e+06* disk./.usagePercent‚16.589113365624964" proc.memory.rssMB‚ 2724.7265625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚42302.38427734375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 290.61328125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 290.61328125 disk.sda.in‚ 0.1953125 network.sent‚ 1.8822369e+07 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.rbd1.in‚0# disk.dm-0.out‚42302.38427734375 disk.sda3.out‚ 290.61328125 disk.drbd2266.out‚0o¸qÆ·‚(:’ ˜ùãÔÅ“û gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚128.121' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.624200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÃÁx¸‚(:í ˜ùãÔˆò… % disk./.usageGB‚247.20920181274414 disk.dm-1.out‚ 291.515625" proc.memory.rssMB‚ 2724.7265625 proc.cpu.threads‚17 network.recv‚ 2.242618e+06 disk.drbd2265.in‚0 disk.sda.out‚ 291.515625 disk.drbd2080.in‚0 network.sent‚ 1.882779e+07 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1167.7109375( disk.nvme0c0n1.out‚42348.87744140625 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚42348.87744140625 disk.dm-0.in‚ 1167.7109375) proc.memory.availableMB‚ 1081.16015625) proc.memory.percent‚16.63041114807129 disk.drbd1716.out‚0 cpu‚25.342445679100518 disk.rbd1.out‚ 29.01953125* disk./.usagePercent‚16.590416955989316 disk.sda3.out‚ 291.515625 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚ 0.1953125 disk.sda1.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 1167.7109375# disk.dm-0.out‚42348.87744140625$ memory_percent‚93.40112209320068‰Ãàú2¹- _wandbruntime‚6139  _runtime‚6139™ ö6ƺÀ & lr-AdamW/pg1‚8.898372156172327e-05 & lr-AdamW/pg2‚8.898372156172327e-05 trainer/global_step‚7899 $ _timestamp‚1.788411036636018e+09 _runtime‚6139.623507425 _step‚646j‡•¾Ë»Å & lr-AdamW/pg2‚8.898372156172327e-05 trainer/global_step‚7899 $ _timestamp‚1.788411036636018e+09 _runtime‚6139.623507425 _step‚646 & lr-AdamW/pg1‚8.898372156172327e-05†ZK%2¼- _wandbruntime‚6139  _runtime‚6139íýÿæ'½àO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114818664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104738166 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6589278535299 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465741952856064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508315998229504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465741278490624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459127652268544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6614282584576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149102441 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚459127652268544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6937512048128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104737728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66991412 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114818666 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚647 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149102441 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614282584576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6613626222080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37746316 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6613626222080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788411036636765e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚7899 _runtime‚6139.624424225 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465741952856064 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114818664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465741278490624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37746752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453688634434179 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚459127670271488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149082403 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6589932973055 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149082403 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6937665186304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚447099355898880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66991414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚508316175307776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459127670271488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ü]C ë'¾åO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465741278490624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459127652268544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6614282584576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149102441 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚459127652268544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6937512048128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104737728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66991412 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114818666 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚647 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149102441 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614282584576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6613626222080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37746316 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6613626222080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788411036636765e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚7899 _runtime‚6139.624424225 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465741952856064 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114818664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465741278490624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37746752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453688634434179 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚459127670271488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149082403 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6589932973055 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149082403 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6937665186304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚447099355898880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66991414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚508316175307776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459127670271488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114818664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104738166 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6589278535299 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465741952856064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508315998229504 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953603 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263933537 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37751270 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66999582 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚7899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149120921 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104750429 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114832682 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501440160859136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938385598464 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114832682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37750849 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚447154260029440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465799125336064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚423 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508378522517504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447154277068800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615102025728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149100857 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66999580 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615102025728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508378703776768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844±!0ö&ÁðM PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104750852 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚6140.316454527 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459184005307392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6938542917632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453745026477027 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459184005307392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465798450957312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149120921 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚7899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149120921 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104750429 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114832682 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501440160859136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938385598464 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114832682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37750849 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚447154260029440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465799125336064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚423 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508378522517504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447154277068800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615102025728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149100857 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66999580 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615102025728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508378703776768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6614445649920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453744354999703 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884110373283703e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114832680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114832680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263953603 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953603 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263933537 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37751270 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66999582 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37973691 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511441643821568 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265527919 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6655268390400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115519466 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6980431095808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115519466 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468600590604288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6654612027904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150008455 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115519464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6980596816896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654612027904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115519464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚475 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511441454160384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6630768312463 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468600590604288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150028493 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37974166 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449844579466240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67399812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504461023064576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6655268390400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105373503 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449844562426880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105373980 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚6176.463160382 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67399814 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504461047004672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150008455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 $ _timestamp‚1.788411073475507e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150028493 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6630113874707 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468599916238848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _step‚651\˜­¼ïÎåO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115519466 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468600590604288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6654612027904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150008455 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115519464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6980596816896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654612027904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115519464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚475 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511441454160384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6630768312463 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468600590604288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150028493 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37974166 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449844579466240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67399812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504461023064576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6655268390400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105373503 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449844562426880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105373980 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚6176.463160382 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67399814 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504461047004672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150008455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 $ _timestamp‚1.788411073475507e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150028493 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6630113874707 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468599916238848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _step‚651 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461945322213888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456475347778703 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456474676301587 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37973691 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511441643821568 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265527919 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6655268390400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115519466 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6980431095808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928‹O“Úš2Ï- _wandbruntime‚6177  _runtime‚6177KAˆkñ&ÐëM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511504147484160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6655431455744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7949 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655431455744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67407980 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462001657249792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449899483596800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580455 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105386148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115533482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115533482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265560389 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6656087831552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511503953641984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265580455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504522697742336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504522673802240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150026909 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6981279839744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462001657249792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115533480 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105386612 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580455 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105386148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115533482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115533482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265560389 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6656087831552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511503953641984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265580455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504522697742336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504522673802240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150026909 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6981279839744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462001657249792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115533480 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105386612 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511504147484160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6655431455744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7949 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655431455744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67407980 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462001657249792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449899483596800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38203730 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507543559947776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7023607686656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452589785994240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150934507 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471458553987072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6671603651871 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464762974156288 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464762956153344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514567008204800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267174811 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116220264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150954545 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464762974156288 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚655 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6670949214115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6695597833728 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106011495 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514567191574528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106011944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459261389646111 _runtime‚6213.112876514 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430IðÙAì'àæO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116220266 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116220266 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452589768954880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚7999 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471458553987072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6671603651871 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464762974156288 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464762956153344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514567008204800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267174811 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116220264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150954545 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464762974156288 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚655 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6670949214115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6695597833728 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106011495 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514567191574528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106011944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459261389646111 _runtime‚6213.112876514 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267154771 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67808214 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7023448257024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464762956153344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6696254196224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150954545 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459260718168995 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38203730 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507543559947776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7023607686656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452589785994240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150934507 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464819327195136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106024227 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7024301260800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452644673085440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6672420087043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459316438734519 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6671765649079 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚471516400832512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116234280 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106024683 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6697073637376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507605234625536 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7999 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150952961 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471516400832512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464819309192192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38207847 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚452644690124800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507605210685440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464819309192192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67816382 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚6213.806408746 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267187241 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150973025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KDDeviceStatsMonitor.on_train_batch_end/inactiv§2üŠ¦e_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150952961 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6697073637376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116234282 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207307 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471515726453760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0L-Òö&ãðM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207307 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471515726453760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38208301 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464819327195136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106024227 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7024301260800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452644673085440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6672420087043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459316438734519 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6671765649079 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚471516400832512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116234280 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106024683 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6697073637376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507605234625536 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7999 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150952961 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471516400832512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464819309192192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38207847 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚452644690124800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507605210685440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464819309192192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67816382 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚6213.806408746 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267187241 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150973025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150952961 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6697073637376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116234282 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0‡©sI2ä- _wandbruntime‚6213  _runtime‚6213ôžàÄVåÐ _step‚657 0loss/SRqqt3/detection_loss‚0.846148994785768 trainer/global_step‚7999 % _timestamp‚1.7884111108205476e+09 epoch‚6 2loss/SRqqt2/assignment_loss‚1.3400848785029917 /loss/FRt3/detection_loss‚1.5716128527771596 0loss/SRqqt1/detection_loss‚2.165462180227041 .loss/FRt4/detection_loss‚2.595112168207401 0loss/SRqqt4/detection_loss‚2.049123133019525 /loss/FBt4/detection_loss‚2.1411253213882446 _runtime‚6213.807923073 /loss/FRt1/assignment_loss‚2.878599128302406 /loss/FRt4/assignment_loss‚2.967453476859302 2loss/SRqqt1/assignment_loss‚1.6536366729065777 2loss/SRqqt3/assignment_loss‚1.0426261793436873 1loss/FBt2/assignment_loss‚0.13809633746859618 0loss/FBt4/assignment_loss‚0.2130480254335063 /loss/FRt2/assignment_loss‚2.845107948002608 /loss/FBt2/detection_loss‚2.1774430349469185 2loss/SRqqt4/assignment_loss‚1.3095706044411173 0loss/FBt3/assignment_loss‚0.1315389355648741 .loss/FRt2/detection_loss‚2.229366527951282 1loss/SRqqt2/detection_loss‚1.6441373108276005 %loss/total_loss‚39.29846805675601 0loss/FRt3/assignment_loss‚1.1790387838385825 0loss/FBt1/assignment_loss‚0.5016640572381826 .loss/FRt1/detection_loss‚2.795114056152456 .loss/FBt1/detection_loss‚2.248664061228434 /loss/FBt3/detection_loss‚0.63469338734174452Žq€[æÕ 1loss/FBt2/assignment_loss‚0.13809633746859618 0loss/FBt4/assignment_loss‚0.2130480254335063 /loss/FRt2/assignment_loss‚2.845107948002608 /loss/FBt2/detection_loss‚2.1774430349469185 2loss/SRqqt4/assignment_loss‚1.3095706044411173 0loss/FBt3/assignment_loss‚0.1315389355648741 .loss/FRt2/detection_loss‚2.229366527951282 1loss/SRqqt2/detection_loss‚1.6441373108276005 %loss/total_loss‚39.29846805675601 0loss/FRt3/assignment_loss‚1.1790387838385825 0loss/FBt1/assignment_loss‚0.5016640572381826 .loss/FRt1/detection_loss‚2.795114056152456 .loss/FBt1/detection_loss‚2.248664061228434 /loss/FBt3/detection_loss‚0.6346933873417445 _step‚657 0loss/SRqqt3/detection_loss‚0.846148994785768 trainer/global_step‚7999 % _timestamp‚1.7884111108205476e+09 epoch‚6 2loss/SRqqt2/assignment_loss‚1.3400848785029917 /loss/FRt3/detection_loss‚1.5716128527771596 0loss/SRqqt1/detection_loss‚2.165462180227041 .loss/FRt4/detection_loss‚2.595112168207401 0loss/SRqqt4/detection_loss‚2.049123133019525 /loss/FBt4/detection_loss‚2.1411253213882446 _runtime‚6213.807923073 /loss/FRt1/assignment_loss‚2.878599128302406 /loss/FRt4/assignment_loss‚2.967453476859302 2loss/SRqqt1/assignment_loss‚1.6536366729065777 2loss/SRqqt3/assignment_loss‚1.0426261793436873‘¨Q%ç‚(:’ òùãÔý̓ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.595800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t;ª€è‚(:õ òùãÔáìÝ  disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.sda.out‚ 295.08203125 disk.sda3.in‚ 0.1953125 disk.drbd2080.out‚0% disk./.usageGB‚247.32455444335938 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚93.40341091156006" proc.memory.rssMB‚ 2724.7265625) proc.memory.percent‚16.63041114807129 network.recv‚ 2.271355e+06 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 1167.7109375 proc.cpu.threads‚17* disk./.usagePercent‚16.598158367817206 disk.sda2.out‚0 disk.sda3.out‚ 295.08203125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 cpu‚25.338197076382897# disk.dm-0.out‚42803.06494140625 disk.drbd2266.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚42803.06494140625 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 29.396484375 disk.dm-0.in‚ 1167.7109375 disk.dm-1.out‚ 295.08203125) proc.memory.availableMB‚ 1080.78515625 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125 network.sent‚ 1.9068722e+07" disk.nvme0c0n1.in‚ 1167.7109375( disk.nvme0c0n1.out‚42803.06494140625ïÿEõ‘é‚(:† úãÔͲ… gpu.0.gpu‚2.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.999' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I¤š|ê‚(:ñ úãÔÕÿ  disk.drbd2265.out‚0% disk./.usageGB‚247.31586837768555 disk.drbd2266.out‚0 disk.rbd1.out‚ 29.396484375 disk.dm-0.in‚ 1167.7109375" proc.memory.rssMB‚ 2724.7265625 proc.cpu.threads‚17 disk.sda2.out‚0' disk.nvme0c0n1.out‚42874.4658203125% disk.nvme0n1.out‚42874.4658203125 disk.sda3.out‚ 295.51953125( proc.memory.availableMB‚ 1081.0234375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd2265.in‚0 network.sent‚ 1.9177706e+07 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 1167.7109375 disk.sda.out‚ 295.51953125 disk.sda3.in‚ 0.1953125" disk.dm-0.out‚42874.4658203125* disk./.usagePercent‚16.597575438661742 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda.in‚ 0.1953125$ memory_percent‚93.40195655822754 cpu‚25.337503775945052 network.recv‚ 2.277721e+06 disk.dm-1.out‚ 295.51953125) proc.memory.percent‚16.63041114807129 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125­Ø˲2ë- _wandbruntime‚6250  _runtime‚6250í¾Åì¿ % lr-AdamW/pg2‚8.34582017544558e-05 trainer/global_step‚8049 % _timestamp‚1.7884111472030165e+09 _runtime‚6250.190445166 _step‚658 % lr-AdamW/pg1‚8.34582017544558e-05Åž*·ÊíÄ _step‚658 % lr-AdamW/pg1‚8.34582017544558e-05 % lr-AdamW/pg2‚8.34582017544558e-05 trainer/global_step‚8049 % _timestamp‚1.7884111472030165e+09 _runtime‚6250.190445166’1A`m2î- _wandbruntime‚6250  _runtime‚6250zz?jä'ïÞO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6736583639552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 6250.1913854 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268781623 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474317191735296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106649444 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517692560182272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7066631169536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8049 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467580626098688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚659 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151880597 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6711784553523 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116921064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462047431513519 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7066463351296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474317866100736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68216614 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510626120771072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737240002048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517692751940608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467580608095744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151880597 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510626096830976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474317191735296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106649444 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517692560182272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7066631169536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8049 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467580626098688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitî„yÇÒor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚659 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151880597 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6711784553523 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116921064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462047431513519 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7066463351296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474317866100736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68216614 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510626120771072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737240002048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517692751940608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467580608095744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151880597 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510626096830976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151899077 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38436845 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462103152079251 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268834159 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116935082 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116935080 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474375038580736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6737403067392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106661625 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474375038580736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834159 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6737403067392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268814093 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6738059443200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814093 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68224782üü“ïö&óðM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116935080 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474375038580736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6737403067392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106661625 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474375038580736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834159 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6737403067392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268814093 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6738059443200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814093 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68224782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68224780 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106662084 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6712600988487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510687771508736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚660 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151899077 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38436845 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462103152079251 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268834159 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116935082 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064”5¨ù‘2ô- _wandbruntime‚6250  _runtime‚6250~~æß]õ× 0loss/FRt1/assignment_loss‚2.5450851608205727 0loss/SRqqt3/detection_loss‚1.070538502519152 1loss/FBt3/assignment_loss‚0.10051750442144089 /loss/FRt2/detection_loss‚2.1072145132788163 trainer/global_step‚8049 _step‚661 2loss/SRqqt4/assignment_loss‚1.2118791021157762 _runtime‚6250.886459836 /loss/FRt2/assignment_loss‚2.866523495940275 1loss/FBt2/assignment_loss‚0.25371483974158765 .loss/FBt4/detection_loss‚2.242475855350494 2loss/SRqqt3/assignment_loss‚1.0355615615304228 2loss/FBt4/assignment_loss‚0.060030513664116825 /loss/FRt3/detection_loss‚1.5337831978836367 1loss/SRqqt4/detection_loss‚1.7243683154469933 /loss/FRt3/assignment_loss‚1.417015561173039 0loss/FRt4/assignment_loss‚2.9868360944092274 0loss/FBt1/assignment_loss‚0.7406931291043257 1loss/SRqqt1/detection_loss‚2.1383534300894964 .loss/FBt1/detection_loss‚2.531358540058136 /loss/FBt3/detection_loss‚0.6304429048770352 epoch‚6 /loss/FRt1/detection_loss‚2.9320877856678433 1loss/SRqqt2/detection_loss‚1.8807358644329584 2loss/SRqqt1/assignment_loss‚1.7556865030810946 2loss/SRqqt2/assignment_loss‚1.3619964220605862 /loss/FRt4/detection_loss‚2.3508517326166234 .loss/FBt2/detection_loss‚2.519801586866379 %loss/total_loss‚39.99755211715003 % _timestamp‚1.7884111478990927e+09Pÿ[SböÜ 1loss/FBt3/assignment_loss‚0.10051750442144089 /loss/FRt2/detection_loss‚2.1072145132788163 trainer/global_step‚8049 _step‚661 2loss/SRqqt4/assignment_loss‚1.2118791021157762 _runtime‚6250.886459836 /loss/FRt2/assignment_loss‚2.866523495940275 1loss/FBt2/assignment_loss‚0.25371483974158765 .loss/FBt4/detection_loss‚2.242475855350494 2loss/SRqqt3/assignment_loss‚1.0355615615304228 2loss/FBt4/assignment_loss‚0.060030513664116825 /loss/FRt3/detection_loss‚1.5337831978836367 1loss/SRqqt4/detection_loss‚1.7243683154469933 /loss/FRt3/assignment_loss‚1.417015561173039 0loss/FRt4/assignment_loss‚2.9868360944092274 0loss/FBt1/assignment_loss‚0.7406931291043257 1loss/SRqqt1/detection_loss‚2.1383534300894964 .loss/FBt1/detection_loss‚2.531358540058136 /loss/FBt3/detection_loss‚0.6304429048770352 epoch‚6 /loss/FRt1/detection_loss‚2.9320877856678433 1loss/SRqqt2/detection_loss‚1.8807358644329584 2loss/SRqqt1/assignment_loss‚1.7556865030810946 2loss/SRqqt2/assignment_loss‚1.3619964220605862 /loss/FRt4/detection_loss‚2.3508517326166234 .loss/FBt2/detection_loss‚2.519801586866379 %loss/total_loss‚39.99755211715003 % _timestamp‚1.7884111478990927e+09 0loss/FRt1/assignment_loss‚2.5450851608205727 0loss/SRqqt3/detection_loss‚1.070538502519152•:ƒ1º÷‚(:… úãÔ‡á… gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?)áÖ{ø‚(:ð úãÔüù¦  disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2724.7265625* disk./.usagePercent‚16.599120700138418 disk.drbd1716.out‚0) proc.memory.percent‚16.63041114807129 disk.nvme0n1.in‚ 1167.7109375 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.drbd2266.in‚0 cpu‚25.336784069142123 proc.cpu.threads‚17 network.recv‚ 2.281464e+06 disk.rbd1.in‚0# disk.dm-0.out‚42954.16455078125 disk.sda.out‚ 296.1484375% disk./.usageGB‚247.33889389038086 disk.sda1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚42954.16455078125 disk.drbd2080.out‚0( proc.memory.availableMB‚ 1081.0234375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚42954.16455078125 disk.drbd1716.in‚0 disk.drbd2080.in‚0 network.sent‚ 1.918305e+07 disk.sda1.out‚0 disk.dm-0.in‚ 1167.7109375 disk.dm-1.out‚ 296.1484375 disk.drbd2265.out‚0$ memory_percent‚93.40195655822754 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.rbd1.out‚ 29.458984375 disk.sda3.out‚ 296.1484375–-ù‚(:’ ŸúãÔ†õ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚129.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýP:Ú~ú‚(:ó ŸúãÔÔ²  disk.drbd2080.out‚0" disk.nvme0c0n1.in‚ 1167.7109375( disk.nvme0c0n1.out‚43005.80908203125 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2724.7265625 disk.dm-0.in‚ 1167.7109375 disk.sda1.out‚0 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2266.in‚0$ memory_percent‚93.40431690216064 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.1953125) proc.memory.availableMB‚ 1080.63671875 network.sent‚ 1.92917e+07* disk./.usagePercent‚16.600103001047295 disk.sda1.in‚0 disk.dm-1.out‚ 296.93359375 disk.sda.out‚ 296.93359375 network.recv‚ 2.287483e+06 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125) proc.memory.percent‚16.63041114807129% disk./.usageGB‚247.35353088378906 disk.rbd1.out‚ 29.458984375# disk.dm-0.out‚43005.80908203125 disk.sda3.in‚ 0.1953125 cpu‚25.336094905311572 disk.rbd0.in‚0& disk.nvme0n1.out‚43005.80908203125 disk.sda3.out‚ 296.93359375 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0gAŸb’û‚(:‡ ®úãÔõó¿ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹‚cÞü‚(:ì ®úãÔž„–  disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.9297629e+07 disk.sda3.in‚ 0.1953125 disk.drbd1716.out‚0! disk.dm-0.out‚43070.544921875 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125" proc.memory.rssMB‚ 2724.726€táSf5625 network.recv‚ 2.292077e+06 disk.drbd2266.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚43070.544921875 disk.rbd1.in‚0 disk.sda.out‚ 297.73046875* disk./.usagePercent‚16.601633926102053 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 29.701171875 disk.dm-0.in‚ 1167.7109375 cpu‚25.33542173577233" disk.nvme0c0n1.in‚ 1167.7109375# memory_percent‚93.4027910232544) proc.memory.availableMB‚ 1080.88671875 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2265.in‚0) proc.memory.percent‚16.63041114807129$ disk./.usageGB‚247.3763427734375 disk.sda3.out‚ 297.73046875 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0$ disk.nvme0n1.out‚43070.544921875 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 297.73046875Æ"yÏ2ý- _wandbruntime‚6286  _runtime‚6286cËfýÇþÁ trainer/global_step‚8099 % _timestamp‚1.7884111838775065e+09 _runtime‚6286.864931383 _step‚662 & lr-AdamW/pg1‚8.163838903509622e-05 & lr-AdamW/pg2‚8.163838903509622e-05nY§ÌÿÆ trainer/global_step‚8099 % _timestamp‚1.7884111838775065e+09 _runtime‚6286.864931383 _step‚662 & lr-AdamW/pg1‚8.163838903509622e-05 & lr-AdamW/pg2‚8.163838903509622e-05–Æjü2€ - _wandbruntime‚6286  _runtime‚6286ZŒ'ç' áO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477176503848960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚458080182010880 trainer/global_step‚8099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428515 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477175829483520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777569445376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7109539424256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152806649 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464833473380927 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270408475 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚470398260038144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38661158 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117621864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107286172 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68625014 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117621864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513708633714176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚458080182010880 trainer/global_step‚8099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428515 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477175829483520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777569445376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7109539424256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152806649 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464833473380927 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270408475 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚470398260038144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38661158 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117621864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107286172 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68625014 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117621864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513708633714176 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚458135086141440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470454631079936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117635882 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152805065 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚520880683786752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6779045249024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477233676328960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520880871337472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270461011 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470454631079936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6754090765859 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚8099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117635880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6779045249024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152825129 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477233676328960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461011 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107298438 _runtime‚6287.554236883 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6778388873216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6753436327895 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152805065 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944R0ÙÜg… ðM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477233001950208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚458135103180800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152825129 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytº*@Ä#es.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117635882 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6778388873216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68633182 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464889193946659 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚458135086141440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470454631079936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117635882 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152805065 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚520880683786752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6779045249024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477233676328960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520880871337472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270461011 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470454631079936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6754090765859 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚8099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117635880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6779045249024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152825129 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477233676328960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461011 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107298438 _runtime‚6287.554236883 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6778388873216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6753436327895 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152805065 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470454613076992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68633180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38665258 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0˜þT{2† - _wandbruntime‚6287  _runtime‚6287L mœU‡ Ï 1loss/SRqqt3/assignment_loss‚1.112479566138063 0loss/FBt4/assignment_loss‚0.2760696449050946 /loss/FBt4/detection_loss‚2.2376326492854526 # _timestamp‚1.78841118456795e+09 /loss/FRt3/detection_loss‚1.6659570616927553 1loss/SRqqt2/detection_loss‚1.8231809415600517 1loss/SRqqt3/detection_loss‚1.0873891179217026 /loss/FRt4/assignment_loss‚2.422883193194866 1loss/FBt2/assignment_loss‚0.19369146786630154 .loss/FRt4/detection_loss‚2.453707315493375 1loss/SRqqt1/detection_loss‚2.7822820417808765 0loss/SRqqt4/detection_loss‚1.731452021041307 .loss/FBt2/detection_loss‚4.465181782841682 trainer/global_step‚8099 _step‚665 0loss/FRt1/assignment_loss‚2.8365305486847374 0loss/FRt2/assignment_loss‚2.8595819708912873 /loss/FRt3/assignment_loss‚1.293595790863037 2loss/SRqqt1/assignment_loss‚1.6185622406051015 2loss/SRqqt2/assignment_loss‚1.3362972145053473 -loss/FRt1/detection_loss‚2.88748964842628 .loss/FBt1/detection_loss‚3.077591905991236 .loss/FBt3/detection_loss‚0.566899233676082 0loss/FBt3/assignment_loss‚0.1310173885423335 2loss/SRqqt4/assignment_loss‚1.5506567172384982 1loss/FBt1/assignment_loss‚0.10234898751089834 /loss/FRt2/detection_loss‚2.3761490825997265 %loss/total_loss‚42.88862753325609 epoch‚6 _runtime‚6287.555338552†jwðZˆ Ô 1loss/SRqqt3/assignment_loss‚1.112479566138063 0loss/FBt4/assignment_loss‚0.2760696449050946 /loss/FBt4/detection_loss‚2.2376326492854526 # _timestamp‚1.78841118456795e+09 /loss/FRt3/detection_loss‚1.6659570616927553 1loss/SRqqt2/detection_loss‚1.8231809415600517 1loss/SRqqt3/detection_loss‚1.0873891179217026 /loss/FRt4/assignment_loss‚2.422883193194866 1loss/FBt2/assignment_loss‚0.19369146786630154 .loss/FRt4/detection_loss‚2.453707315493375 1loss/SRqqt1/detection_loss‚2.7822820417808765 0loss/SRqqt4/detection_loss‚1.731452021041307 .loss/FBt2/detection_loss‚4.465181782841682 trainer/global_step‚8099 _step‚665 0loss/FRt1/assignment_loss‚2.8365305486847374 0loss/FRt2/assignment_loss‚2.8595819708912873 /loss/FRt3/assignment_loss‚1.293595790863037 2loss/SRqqt1/assignment_loss‚1.6185622406051015 2loss/SRqqt2/assignment_loss‚1.3362972145053473 -loss/FRt1/detection_loss‚2.88748964842628 .loss/FBt1/detection_loss‚3.077591905991236 .loss/FBt3/detection_loss‚0.566899233676082 0loss/FBt3/assignment_loss‚0.1310173885423335 2loss/SRqqt4/assignment_loss‚1.5506567172384982 1loss/FBt1/assignment_loss‚0.10234898751089834 /loss/FRt2/detection_loss‚2.3761490825997265 %loss/total_loss‚42.88862753325609 epoch‚6 _runtime‚6287.555338552™5@Õ’‰ ‚(:‡ ½úãÔà¸Ä gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÐhƒpŠ ‚(:å ½úãÔÞÿÕ  network.sent‚ 1.9304094e+07 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 29.701171875! disk.dm-0.out‚43148.666015625" proc.memory.rssMB‚ 2724.7265625% disk./.usageGB‚247.40972900390625 disk.sda.out‚ 298.03125$ disk.nvme0n1.out‚43148.666015625 disk.dm-0.in‚ 1167.7109375 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.1953125 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 298.03125* disk./.usagePercent‚16.603874504041723 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.dm-1.in‚ 0.1953125) proc.memory.percent‚16.63041114807129 network.recv‚ 2.297048e+06 disk.rbd1.in‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 298.03125 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 1167.7109375$ memory_percent‚93.40472221374512( proc.memory.availableMB‚ 1080.5703125 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1167.7109375& disk.nvme0c0n1.out‚43148.666015625 cpu‚25.334732374776774¨ Þ’‹ ‚(:‡ ÌúãÔ±’§ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ü|fwŒ ‚(:ì ÌúãÔµ­­ * disk./.usagePercent‚16.607118631515615 disk.sda2.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚43274.798828125 proc.cpu.threads‚17 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 29.763671875( proc.memory.availableMB‚ 1080.5703125 network.recv‚ 2.30261e+06 disk.drbd2266.in‚0 disk.drbd2080.out‚0& disk.nvme0c0n1.out‚43274.798828125$ memory_percent‚93.40472221374512 disk.sda3.out‚ 298.66796875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 298.66796875 disk.sda1.in‚0 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2724.7265625 disk.dm-1.out‚ 298.66796875 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.sda.in‚ 0.1953125 network.sent‚ 1.9412245e+07% disk./.usageGB‚247.45806884765625 disk.dm-0.in‚ 1167.7109375! disk.dm-0.out‚43274.798828125) proc.memory.percent‚16.63041114807129 cpu‚25.33403018865579¢T™ô2 - _wandbruntime‚6323  _runtime‚6323ߎlÆŽ À trainer/global_step‚8149 % _timestamp‚1.7884112207888587e+09 _runtime‚ 6323.77628828 _step‚666 & lr-AdamW/pg1‚7.983022394468105e-05 & lr-AdamW/pg2‚7.983022394468105e-05¨1¹Ë Å & lr-AdamW/pg2‚7.983022394468105e-05 trainer/global_step‚8149 % _timestamp‚1.7884112207888587e+09 _runtime‚ 6323.77628828 _step‚666 & lr-AdamW/pg1‚7.983022394468105e-05š]|j2 - _wandbruntime‚6323  _runtime‚6323©—dÏç'‘ áO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460825405578240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480035141597184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819211613696 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107922052 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118322664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6794109670095 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚8149 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118322664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7152772940800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272055367 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118322666 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38888638 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467619515248335 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819211613696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818555251200 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153712663 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523943779914240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7152609316864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473215911980544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚473215929983488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523943967478272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚460825388538880 _runtime‚6323.777253382 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 272035327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516791170597376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69033412 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚516791194537472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473215911980544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69033414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480034467231744 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107921606 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚467618843771219 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚473215929983488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153732701 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 % _timestamp‚1.7884112207895641e+09 _step‚667zøúG’ æO _runtime‚6323.777253382 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 272035327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516791170597376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69033412 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚516791194537472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473215911980544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69033414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480034467231744 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107921606 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚467618843771219 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚473215929983488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153732701 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 % _timestamp‚1.7884112207895641e+09 _step‚667 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460825405578240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480035141597184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819211613696 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107922052 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.ç'rî¥ on_train_batch_start/allocation.large_pool.freed‚ 118322664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6794109670095 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚8149 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118322664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7152772940800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272055367 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118322666 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38888638 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467619515248335 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819211613696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818555251200 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153712663 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523943779914240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7152609316864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473215911980544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚473215929983488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523943967478272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚460825388538880›8Ó'2“ - _wandbruntime‚6324  _runtime‚6324òðN=ð&” êM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118336680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69041580 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 174967808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480092314077184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153731117 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460880309708800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚473272265019392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118336682 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272087863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38893146 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524006484883968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38892703 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467675235814067 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6794926105267 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6820031054848 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6794271667303 _step‚668 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087863 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473272265019392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7153488581120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467674564336743 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153751181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6820031054848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118336682 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚516852845275136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚8149 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107934283 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7153639608832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚6324.467235522 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272067797 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6819374679040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 $ _timestamp‚1.788411221479008e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473272283022336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480092314077184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516852821335040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151027712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153731117 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69041582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473272283022336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚460880292669440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480091639698432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819374679040 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272087863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38893146 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524006484883968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38892703 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467675235814067 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6794926105267 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6820031054848 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6794271667303 _step‚668 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087863 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473272265019392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7153488581120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467674564336743 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153751181 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6820031054848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118336682 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚516852845275136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚8149 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107934283 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7153639608832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚6324.467235522 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272067797 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6819374679040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 $ _timestamp‚1.788411221479008e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473272283022336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480092314077184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516852821335040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151027712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153731117 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69041582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473272283022336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚460880292669440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480091639698432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819374679040 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0œ…¡xø2– - _wandbruntime‚6324  _runtime‚6324~šÃ|_— Ù 2loss/SRqqt1/assignment_loss‚1.6515001866125292 1loss/FBt2/assignment_loss‚0.13994026820485791 1loss/SRqqt1/detection_loss‚2.6360506684549394 %loss/total_loss‚41.03343669145693 _runtime‚6324.468492404 1loss/FBt3/assignment_loss‚0.11802449325891101 .loss/FRt1/detection_loss‚3.045776572227478 /loss/FBt1/detection_loss‚2.9069539989743913 % _timestamp‚1.7884112214811387e+09 2loss/SRqqt4/assignment_loss‚1.1552515139066897 1loss/FBt1/assignment_loss‚0.45843453206388013 /loss/FRt2/detection_loss‚2.5431676588952685 .loss/FRt4/detection_loss‚2.334650227711314 /loss/FBt4/detection_loss‚2.2656449344423084 0loss/FRt1/assignment_loss‚2.9551918077468873 2loss/SRqqt2/assignment_loss‚1.3404510477557778 _step‚669 0loss/FRt3/assignment_loss‚1.3063746481030076 2loss/SRqqt3/assignment_loss‚0.9151563942027481 1loss/SRqqt3/detection_loss‚0.9188660852395106 /loss/FBt3/detection_loss‚0.5249342300124805 0loss/FRt2/assignment_loss‚2.2585423231124877 /loss/FRt4/assignment_loss‚3.057463465701966 1loss/FBt4/assignment_loss‚0.18805782992341039 /loss/FRt3/detection_loss‚1.6741396818241034 1loss/SRqqt2/detection_loss‚1.9746470153331757 1loss/SRqqt4/detection_loss‚2.0861473066169163 .loss/FBt2/detection_loss‚2.578069801131884 trainer/global_step‚8149 epoch‚6Zß%dd˜ Þ 0loss/FRt2/assignment_loss‚2.2585423231124877 /loss/FRt4/assignment_loss‚3.057463465701966 1loss/FBt4/assignment_loss‚0.18805782992341039 /loss/FRt3/detection_loss‚1.6741396818241034 1loss/SRqqt2/detection_loss‚1.9746470153331757 1loss/SRqqt4/detection_loss‚2.0861473066169163 .loss/FBt2/detection_loss‚2.578069801131884 trainer/global_step‚8149 epoch‚6 2loss/SRqqt1/assignment_loss‚1.6515001866125292 1loss/FBt2/assignment_loss‚0.13994026820485791 1loss/SRqqt1/detection_loss‚2.6360506684549394 %loss/total_loss‚41.03343669145693 _runtime‚6324.468492404 1loss/FBt3/assignment_loss‚0.11802449325891101 .loss/FRt1/detection_loss‚3.045776572227478 /loss/FBt1/detection_loss‚2.9069539989743913 % _timestamp‚1.7884112214811387e+09 2loss/SRqqt4/assignment_loss‚1.1552515139066897 1loss/FBt1/assignment_loss‚0.45843453206388013 /loss/FRt2/detection_loss‚2.5431676588952685 .loss/FRt4/detection_loss‚2.334650227711314 /loss/FBt4/detection_loss‚2.2656449344423084 0loss/FRt1/assignment_loss‚2.9551918077468873 2loss/SRqqt2/assignment_loss‚1.3404510477557778 _step‚669 0loss/FRt3/assignment_loss‚1.3063746481030076 2loss/SRqqt3/assignment_loss‚0.9151563942027481 1loss/SRqqt3/detection_loss‚0.9188660852395106 /loss/FBt3/detection_loss‚0.5249342300124805™\{™ ‚(:… ÛúãÔºêš gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.32' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü]¯:sš ‚(:è ÛúãÔØʈ  network.recv‚ 2.306779e+06 disk.drbd2080.in‚0 disk.rbd1.out‚ 29.763671875" proc.memory.rssMB‚ 2724.7265625 cpu‚25.33335396826124 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 299.2265625 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1167.7109375 disk.rbd0.in‚0 disk.drbd2266.out‚0 network.sent‚ 1.9418199e+07" disk.nvme0c0n1.in‚ 1167.7109375& disk.nvme0c0n1.out‚43351.962890625$ memory_percent‚93.40505599975586 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125% disk./.usageGB‚247.47913360595703 disk.dm-1.out‚ 299.2265625 disk.drbd2080.out‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 1080.515625 disk.sda.out‚ 299.2265625 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1167.7109375) proc.memory.percent‚16.63041114807129) disk./.usagePercent‚16.60853230511969 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚43351.962890625! disk.dm-0.out‚43351.962890625Š½Œ=› ‚(:’ êúãÔ˜Š gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.865' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.973000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ1Ùy}œ ‚(:ò êúãÔ¼Çß ' proc.memory.availableMB‚ 1080.265625 disk.nvme0n1.in‚ 1167.7109375 disk.rbd1.in‚0* disk./.usagePercent‚16.609858424147056& disk.nvme0n1.out‚43412.57080078125 disk.sda3.out‚ 299.84765625 disk.rbd0.in‚0 disk.rbd1.out‚ 29.9521484375$ memory_percent‚93.40658187866211) proc.memory.percent‚16.63041114807129 disk.dm-0.in‚ 1167.7109375 disk.sda.in‚ 0.1953125 network.sent‚ 1.952683e+07 disk.sda3.in‚ 0.1953125 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚43412.57080078125 disk.drbd1716.in‚0 network.recv‚ 2.313144e+06 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2724.7265625 proc.cpu.threads‚17# disk.dm-0.out‚43412.57080078125 disk.dm-1.out‚ 299.84765625 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 cpu‚25.33261084306845% disk./.usageGB‚247.49889373779297 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 299.84765625 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1167.7109375û]@í ‚(:’ ùúãÔ ÝÙ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.425' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.885000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸}ž ‚(:ô ùúãÔïÍç $ memory_percent‚93.40503215789795 network.sent‚ 1.9533125e+07 network.recv‚ 2.31777e+06 disk.dm-1.out‚ 300.35546875 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0% disk./.usageGB‚247.51655197143555 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1167.7109375 cpu‚25.33197724773923 disk.drbd1716.in‚0* disk./.usagePercent‚16.611043483023057 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1167.7109375# disk.dm-0.out‚43482.26220703125 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 300.35546875 disk.drbd2266.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚43482.26220703125 disk.dm-0.in‚ 1167.7109375 disk.drbd2080.in‚0) proc.memory.availableMB‚ 1080.51953125" proc.memory.rssMB‚ 2724.7265625) proc.memory.percent‚16.63041114807129 disk.sda.in‚ 0.1953125 disk.drbd1716.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚43482.26220703125 disk.rbd1.out‚ 29.9521484375 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 300.35546875 disk.drbd2266.out‚0w Òé2Ÿ - _wandbruntime‚6360  _runtime‚6360Lp óÆ  À trainer/global_step‚8199 $ _timestamp‚1.788411257499089e+09 _runtime‚6360.486535892 _step‚670 & lr-AdamW/pg1‚7.803408330210494e-05 & lr-AdamW/pg2‚7.803408330210494e-05"ÍÌ)Ë¡ Å $ _timestamp‚1.788411257499089e+09 _runtime‚6360.486535892 _step‚670 & lr-AdamW/pg1‚7.803408330210494e-05 & lr-AdamW/pg2‚7.803408330210494e-05 trainer/global_step‚8199ž3›/å2¢ - _wandbruntime‚6360  _runtime‚6360Çfg££ áO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚476033581925888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6859541057024 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108559124 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476033563922944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108558663 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚527069388602880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚482893779345408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚482893104979968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMŽ£áøD"onitor.on_train_batch_start/active_bytes.large_pool.allocated‚476033581925888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚482893779345408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884112574998176e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119023464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6860197419520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚671 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273682219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6834945009503 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119023466 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39116851 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470404885638627 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273662179 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39117310 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7195681122304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚8199 _runtime‚6360.487487007 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154638715 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6859541057024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7195853134848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119023464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69441812 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470405557115743 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚527069584555520 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119023466 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154658753 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚476033563922944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6860197419520 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154638715 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482893104979968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519873731420672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6834290571747 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69441814 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482893104979968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519873731420672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6834290571747 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69441814 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476033563922944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108558663 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚527069388602880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚482893779345408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚482893104979968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚476033581925888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚482893779345408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884112574998176e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119023464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6860197419520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚671 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273682219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6834945009503 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119023466 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39116851 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470404885638627 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273662179 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39117310 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7195681122304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚8199 _runtime‚6360.487487007 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154638715 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6859541057024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7195853134848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119023464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69441812 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470405557115743 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚527069584555520 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119023466 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154658753 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚476033563922944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6860197419520 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154638715 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401ŸQ%í2¥ - _wandbruntime‚6361  _runtime‚6361MLœÙñ&¦ ëM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476089934964736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154657169 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519935382158336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6860360484864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚527132092625408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119037482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476089934964736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119037482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚482950951825408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273694649 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6835761444675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69449980 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476089916961792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154677233 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108571313 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6860360484864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7196710467072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714715 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519935358218240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚482950951825408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470461277681475òzeŠ§ ðM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119037482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476089934964736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119037482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚482950951825408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273694649 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6835761444675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69449980 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476089916961792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154677233 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108571313 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6860360484864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7196710467072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714715 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519935358218240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚482950951825408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470461277681475 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476089934964736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154657169 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519935382158336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6860360484864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚527132092625408 š¥i“2¨ - _wandbruntime‚6361  _runtime‚6361‹EFé`© Ú /loss/FRt2/detection_loss‚2.3640736388049013 2loss/SRqqt3/assignment_loss‚1.1595860135837877 1loss/FBt2/assignment_loss‚0.39066089534511167 % _timestamp‚1.7884112581938648e+09 0loss/FRt2/assignment_loss‚2.8544036050637565 2loss/SRqqt4/assignment_loss‚1.3875694523929902 2loss/FBt4/assignment_loss‚0.011493849800899625 1loss/SRqqt3/detection_loss‚1.0315512080259364 /loss/FBt1/detection_loss‚2.0798568189144135 &loss/total_loss‚42.184718304870216 _runtime‚6361.181235447 0loss/FRt4/assignment_loss‚2.7364617099887445 2loss/SRqqt1/assignment_loss‚1.5239802114665508 0loss/FBt1/assignment_loss‚0.2258982581901364 /loss/FBt4/detection_loss‚2.8214324166377387 .loss/FRt1/detection_loss‚3.210976535311112 /loss/FRt4/detection_loss‚2.7653051191254665 _step‚673 1loss/SRqqt1/detection_loss‚2.4551039934158325 1loss/SRqqt2/detection_loss‚1.9008115126440923 0loss/FBt3/assignment_loss‚0.0974597335533298 .loss/FBt2/detection_loss‚2.933650334676107 trainer/global_step‚8199 0loss/FRt1/assignment_loss‚3.4266854455837836 0loss/FRt3/assignment_loss‚1.3201303509778755 2loss/SRqqt2/assignment_loss‚1.4132780183572322 /loss/FRt3/detection_loss‚1.7217800237758214 0loss/SRqqt4/detection_loss‚1.827409899915042 /loss/FBt3/detection_loss‚0.5251592593195551 epoch‚6àqõeª ß 0loss/SRqqt4/detection_loss‚1.827409899915042 /loss/FBt3/detection_loss‚0.5251592593195551 epoch‚6 /loss/FRt2/detection_loss‚2.3640736388049013 2loss/SRqqt3/assignment_loss‚1.1595860135837877 1loss/FBt2/assignment_loss‚0.39066089534511167 % _timestamp‚1.7884112581938648e+09 0loss/FRt2/assignment_loss‚2.8544036050637565 2loss/SRqqt4/assignment_loss‚1.3875694523929902 2loss/FBt4/assignment_loss‚0.011493849800899625 1loss/SRqqt3/detection_loss‚1.0315512080259364 /loss/FBt1/detection_loss‚2.0798568189144135 &loss/total_loss‚42.184718304870216 _runtime‚6361.181235447 0loss/FRt4/assignment_loss‚2.7364617099887445 2loss/SRqqt1/assignment_loss‚1.5239802114665508 0loss/FBt1/assignment_loss‚0.2258982581901364 /loss/FBt4/detection_loss‚2.8214324166377387 .loss/FRt1/detection_loss‚3.210976535311112 /loss/FRt4/detection_loss‚2.7653051191254665 _step‚673 1loss/SRqqt1/detection_loss‚2.4551039934158325 1loss/SRqqt2/detection_loss‚1.9008115126440923 0loss/FBt3/assignment_loss‚0.0974597335533298 .loss/FBt2/detection_loss‚2.933650334676107 trainer/global_step‚8199 0loss/FRt1/assignment_loss‚3.4266854455837836 0loss/FRt3/assignment_loss‚1.3201303509778755 2loss/SRqqt2/assignment_loss‚1.4132780183572322 /loss/FRt3/detection_loss‚1.7217800237758214¡È-¤û’« ‚(:‡ ˆûãÔýï‹ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.381' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø-õ¹d¬ ‚(:Ù ˆûãÔÉ≠ disk.drbd2265.in‚0" proc.memory.rssMB‚ 2724.7265625 network.sent‚ 1.9537501e+07 disk.sda1.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 1080.453125) proc.memory.percent‚16.63041114807129 disk.nvme0n1.in‚ 1167.7109375 disk.sda.out‚ 300.94140625 disk.rbd0.out‚0 network.recv‚ 2.320677e+06" disk.nvme0c0n1.in‚ 1167.7109375 disk.dm-0.in‚ 1167.7109375 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0$ memory_percent‚93.40543746948242 disk.drbd1716.out‚0 disk.rbd1.out‚ 30.0146484375 disk.nvme0n1.out‚43546.25 disk.dm-0.out‚43546.25 disk.dm-1.out‚ 300.94140625 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.rbd0.in‚0* disk./.usagePercent‚16.612079545514124 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 cpu‚25.331291588074862 proc.cpu.threads‚17 disk.sda2.out‚0 disk.nvme0c0n1.out‚43546.25 disk.rbd1.in‚0 disk.sda3.out‚ 300.94140625% disk./.usageGB‚247.53199005126953 disk.sda1.out‚0 disk.sda3.in‚ 0.1953125ÇÆlö­ ‚(:’ —ûãÔ„é gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚156.519' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.303800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ÞÚH® ‚(:ô —ûãÔˆÈÞ  disk.rbd1.in‚0# disk.dm-0.out‚43613.07373046875 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1080.44921875" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.in‚0 disk.rbd0.out‚0$ memory_percent‚93.40546131134033" proc.memory.rssMB‚ 2724.7265625) proc.memory.percent‚16.63041114807129 cpu‚25.330589460516368 network.sent‚ 1.9646255e+07 disk.sda2.out‚0& disk.nvme0n1.out‚43613.07373046875 disk.dm-1.out‚ 301.36328125 disk.sda3.out‚ 301.36328125 network.recv‚ 2.326916e+06 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚43613.07373046875 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.sda.out‚ 301.36328125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 1167.7109375 disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0$ disk./.usageGB‚247.5465202331543 disk.rbd1.out‚ 30.0146484375) disk./.usagePercent‚16.61305467821205 disk.sda1.out‚0`ÏÎ/2¯ - _wandbruntime‚6397  _runtime‚6397‘*0Ç° Á _step‚674 & lr-AdamW/pg1‚7.625034142038573e-05 & lr-AdamW/pg2‚7.625034142038573e-05 trainer/global_step‚8249 % _timestamp‚1.7884112944311295e+09 _runtime‚6397.418569641tž•Ì± Æ _runtime‚6397.418569641 _step‚674 & lr-AdamW/pg1‚7.625034142038573e-05 & lr-AdamW/pg2‚7.625034142038573e-05 trainer/global_step‚8249 % _timestamp‚1.7884112944311295e+09¢Ÿç=É2² - _wandbruntime‚6397  _runtime‚6397CûRþæ'³ àO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119724264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485751742728192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚473190927506035 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275289031 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530195138712064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473191598983151 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239041195008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119724264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478851215865344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478851233868288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69850214 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6900526862848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚522956244363776 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6901183225344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478851215865344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119724266 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275309071 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530195309498880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485752417093632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109196751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚522956268303872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0›¨àL&´ åO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155584805 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155564767 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155564767 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466315818634240 _runtime‚6397.419537748 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485751742728192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478851233868288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485752417093632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119724266 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109197201 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6875125911155 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69850212 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚8249 $ _timestamp‚1.788411294431875e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7238894348288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 170786816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 146846720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6875780348911 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155584805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6900526862848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39346987 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485751742728192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚473190927506035 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275289031 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530195138712064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473191598983151 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239041195008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119724264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478851215865344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478851233868288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69850214 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6900526862848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚522956244363776 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6901183225344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478851215865344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119724266 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275309071 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530195309498880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485752417093632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109196751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚522956268303872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119724264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JŸMonitor.on_train_batch_start/active.all.allocated‚ 275309071 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6901183225344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39346539 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚466315801594880£ÔKw?2µ - _wandbruntime‚6398  _runtime‚6398™¦Ž:ñ&¶ ëM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530257835442688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚466370705725440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6901346290688 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341567 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478907568904192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39351500 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523017895101440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119738280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6901346290688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7239916401152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚473246648071559 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155603285 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530257649989120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6902002666496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275321501 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155583221 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109209432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523017919041536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚8249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155603285 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485809589573632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69858380 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6902002666496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155583221 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚6398.112143952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478907586907136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478907586907136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341567 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119738280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69858382 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275321501 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109209882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39351052 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚485809589573632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119738282 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466370722764800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119738282 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7239754887680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473247319548883 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048`“‰ö&· ðM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119738282 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7239754887680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473247319548883 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530257835442688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚466370705725440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6901346290688 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341567 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478907568904192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39351500 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523017895101440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119738280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6901346290688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7239916401152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚473246648071559 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155603285 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530257649989120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6902002666496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275321501 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155583221 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109209432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523017919041536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚8249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155603285 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485809589573632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69858380 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6902002666496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155583221 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚6398.112143952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478907586907136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478907586907136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341567 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119738280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69858382 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275321501 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109209882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39351052 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚485809589573632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119738282 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466370722764800¤V¿;A2¸ - _wandbruntime‚6398  _runtime‚6398•IY¹ Ó 0loss/FRt4/assignment_loss‚2.5130022741713613 .loss/FRt1/detection_loss‚2.781677939675071 1loss/SRqqt3/detection_loss‚0.9478322274982929 /loss/FBt3/detection_loss‚0.6094597498760432 _step‚677 /loss/FRt1/assignment_loss‚3.793848046750733 2loss/SRqqt2/assignment_loss‚1.5601118375049081 0loss/SRqqt3/assignment_loss‚0.77257049655697 /loss/FRt3/detection_loss‚1.8083679393678904 &loss/total_loss‚41.521793869159055 _runtime‚6398.113268174 2loss/SRqqt1/assignment_loss‚1.9583965499658842 1loss/FBt3/assignment_loss‚0.15697176686291958 /loss/FRt2/detection_loss‚2.2533135839073584 0loss/SRqqt4/detection_loss‚1.637670451177741 1loss/FBt2/assignment_loss‚0.14764390426820942 .loss/FBt4/detection_loss‚2.068641444047292 epoch‚6 trainer/global_step‚8249 /loss/FBt1/detection_loss‚2.5642674639821053 % _timestamp‚1.7884112951258903e+09 2loss/SRqqt4/assignment_loss‚1.2759880291403465 .loss/FRt4/detection_loss‚2.293230026960373 1loss/FBt4/assignment_loss‚0.10509563645529044 0loss/SRqqt2/detection_loss‚2.036890528922857 0loss/FRt2/assignment_loss‚3.6014863327145576 /loss/FRt3/assignment_loss‚1.621099909543991 0loss/FBt1/assignment_loss‚0.5524370111816088 1loss/SRqqt1/detection_loss‚2.2083179306339575 .loss/FBt2/detection_loss‚2.253472787993295óH¦ò^º Ø 1loss/FBt4/assignment_loss‚0.10509563645529044 0loss/SRqqt2/detection_loss‚2.036890528922857 0loss/FRt2/assignment_loss‚3.6014863327145576 /loss/FRt3/assignment_loss‚1.621099909543991 0loss/FBt1/assignment_loss‚0.5524370111816088 1loss/SRqqt1/detection_loss‚2.2083179306339575 .loss/FBt2/detection_loss‚2.253472787993295 0loss/FRt4/assignment_loss‚2.5130022741713613 .loss/FRt1/detection_loss‚2.781677939675071 1loss/SRqqt3/detection_loss‚0.9478322274982929 /loss/FBt3/detection_loss‚0.6094597498760432 _step‚677 /loss/FRt1/assignment_loss‚3.793848046750733 2loss/SRqqt2/assignment_loss‚1.5601118375049081 0loss/SRqqt3/assignment_loss‚0.77257049655697 /loss/FRt3/detection_loss‚1.8083679393678904 &loss/total_loss‚41.521793869159055 _runtime‚6398.113268174 2loss/SRqqt1/assignment_loss‚1.9583965499658842 1loss/FBt3/assignment_loss‚0.15697176686291958 /loss/FRt2/detection_loss‚2.2533135839073584 0loss/SRqqt4/detection_loss‚1.637670451177741 1loss/FBt2/assignment_loss‚0.14764390426820942 .loss/FBt4/detection_loss‚2.068641444047292 epoch‚6 trainer/global_step‚8249 /loss/FBt1/detection_loss‚2.5642674639821053 % _timestamp‚1.7884112951258903e+09 2loss/SRqqt4/assignment_loss‚1.2759880291403465 .loss/FRt4/detection_loss‚2.293230026960373¥òrýÝ’» ‚(:‡ ¦ûãÔÉäž gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï_ ,n¼ ‚(:ã ¦ûãÔ™’  disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 30.2890625 disk.nvme0n1.in‚ 1167.7109375 cpu‚25.32989278988778 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚43686.634765625 disk.drbd1716.in‚0 disk.dm-1.out‚ 302.03125" proc.memory.rssMB‚ 2724.7265625 network.sent‚ 1.9652132e+07! disk.dm-0.out‚43686.634765625) proc.memory.percent‚16.63041114807129 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1167.7109375 proc.cpu.threads‚17 network.recv‚ 2.331277e+06 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125$ memory_percent‚93.40565204620361% disk./.usageGB‚247.56398010253906 disk.sda.out‚ 302.03125 disk.sda3.out‚ 302.03125$ disk.nvme0n1.out‚43686.634765625* disk./.usagePercent‚16.614226424696266 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 1167.7109375) proc.memory.availableMB‚ 1080.417968750øO½ ‚(:’ µûãÔŽˆþ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.168' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.233599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çÈËÿg¾ ‚(:Ü µûãÔÐÕ» ! disk.nvme0c0n1.out‚ 43749.4375% disk./.usageGB‚247.57724380493164 disk.sda2.in‚0" proc.memory.rssMB‚ 2724.7265625 disk.drbd2265.out‚0 disk.drbd2266.in‚0$ memory_percent‚93.40567588806152 disk.nvme0n1.out‚ 43749.4375 network.sent‚ 1.9761108e+07 network.recv‚ 2.338017e+06) disk./.usagePercent‚16.61511656289282 disk.rbd1.out‚ 30.2890625 disk.nvme0n1.in‚ 1167.7109375 disk.sda1.in‚0( proc.memory.availableMB‚ 1080.4140625 disk.sda3.out‚ 302.58984375" disk.nvme0c0n1.in‚ 1167.7109375 disk.dm-0.in‚ 1167.7109375 disk.drbd1716.in‚0 disk.sda.out‚ 302.58984375 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 302.58984375 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0) proc.memory.percent‚16.63041114807129 cpu‚25.329238586500093 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1953125 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-0.out‚ 43749.4375 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda2.out‚0)ÇžÚ2¿ - _wandbruntime‚6434  _runtime‚6434:ËÑÅÀ ¿ % _timestamp‚1.7884113311028705e+09 _runtime‚6434.090309602 _step‚678 % lr-AdamW/pg1‚7.44793700286581e-05 % lr-AdamW/pg2‚7.44793700286581e-05 trainer/global_step‚8299ʦíÊÁ Ä _runtime‚6434.090309602 _step‚678 % lr-AdamW/pg1‚7.44793700286581e-05 % lr-AdamW/pg2‚7.44793700286581e-05 trainer/global_step‚8299 % _timestamp‚1.7884113311028705e+09¦Z*Ù2 - _wandbruntime‚6434  _runtime‚6434áŠqýøà áO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚475977640850559 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120425066 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109833098 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚526038805187072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488610380476416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚481668867807744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚8299 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156510857 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156490819 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39574486 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚469061008122880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488611054841856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70258612 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚488610380476416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚481668885810688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276915883 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156510857 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6942169031168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚481668885810688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488611054841856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276935923 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109833559 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚679 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481668867807744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120425066 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533320854031872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096¤óÄì'Ä æO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚481668885810688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488611054841856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276935923 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109833559 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚679 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481668867807744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120425066 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533320854031872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚475977640850559 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120425066 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109833098 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚526038805187072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488610380476416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚481668867807744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚8299 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156510857 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156490819 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39574486 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚469061008122880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488611054841856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70258612 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚488610380476416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚481668885810688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276915883 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156510857 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6942169031168§¡ëeÃ2Å - _wandbruntime‚6434  _runtime‚6434Ùo$ñ&Æ ëM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚476032689938967 _runtime‚6434.787818673 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942988472320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109846231 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276968419 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6917432123491 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533383369671168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526100431984640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚469115929292800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8299 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476033361416291 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916777685527 _step‚680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156509273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884113317997723e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948353 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120439082 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481725220846592 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276948353 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488667552943104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481725238849536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481725220846592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968419 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70266780 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533383182120448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526100455924736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481725238849536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120439080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39579449 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109845781 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156509273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚469115912253440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156529337 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6942988472320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096ÐÐtÈßÇ ðM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6917432123491 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533383369671168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526100431984640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚469115929292800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8299 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476033361416291 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916777685527 _step‚680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156509273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884113317997723e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948353 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120439082 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481725220846592 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276948353 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488667552943104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481725238849536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481725220846592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968419 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70266780 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533383182120448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526100455924736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481725238849536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120439080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39579449 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109845781 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156509273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚469115912253440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156529337 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6942988472320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 S?DeviceStatsMonitor.on_train_batch_end!N7/requested_bytes.all.freed‚476032689938967 _runtime‚6434.787818673 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942988472320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109846231 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276968419 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048¨#)½2È - _wandbruntime‚6434  _runtime‚6434¿uÙ']É × 0loss/FRt1/assignment_loss‚3.5258900948933194 0loss/FRt3/assignment_loss‚1.7365024089813232 /loss/FBt2/detection_loss‚3.8525114258130393 &loss/total_loss‚41.794125871339354 2loss/SRqqt3/assignment_loss‚0.9844027922550838 1loss/FBt1/assignment_loss‚0.04974662529171476 trainer/global_step‚8299 1loss/FBt2/assignment_loss‚0.30316011762867373 .loss/FRt2/detection_loss‚2.043135495049586 1loss/SRqqt3/detection_loss‚0.8803672684232394 1loss/SRqqt4/detection_loss‚1.8019688190973324 /loss/FBt3/detection_loss‚0.5493761141072301 /loss/FRt1/detection_loss‚2.9695202984980176 /loss/FRt3/detection_loss‚1.6919726165166746 epoch‚6 _runtime‚6434.788901006 _step‚681 % _timestamp‚1.7884113318015423e+09 /loss/FRt2/assignment_loss‚3.255725917951116 /loss/FRt4/detection_loss‚2.3525536554780873 .loss/SRqqt1/detection_loss‚2.0933394767344 1loss/SRqqt2/detection_loss‚1.7436008365715252 /loss/FRt4/assignment_loss‚2.815054405819286 2loss/SRqqt1/assignment_loss‚1.3642262036446482 2loss/SRqqt2/assignment_loss‚1.3616326444727533 2loss/SRqqt4/assignment_loss‚1.3320311439911956 0loss/FBt4/assignment_loss‚0.2915819893325014 /loss/FBt1/detection_loss‚2.6448995045253207 /loss/FBt4/detection_loss‚2.0677769354411533 1loss/FBt3/assignment_loss‚0.08314908082213368œwbÊ Ü 0loss/FRt1/assignment_loss‚3.5258900948933194 0loss/FRt3/assignment_loss‚1.7365024089813232 /loss/FBt2/detection_loss‚3.8525114258130393 &loss/total_loss‚41.794125871339354 2loss/SRqqt3/assignment_loss‚0.9844027922550838 1loss/FBt1/assignment_loss‚0.04974662529171476 trainer/global_step‚8299 1loss/FBt2/assignment_loss‚0.30316011762867373 .loss/FRt2/detection_loss‚2.043135495049586 1loss/SRqqt3/detection_loss‚0.8803672684232394 1loss/SRqqt4/detection_loss‚1.8019688190973324 /loss/FBt3/detection_loss‚0.5493761141072301 /loss/FRt1/detection_loss‚2.9695202984980176 /loss/FRt3/detection_loss‚1.6919726165166746 epoch‚6 _runtime‚6434.788901006 _step‚681 % _timestamp‚1.7884113318015423e+09 /loss/FRt2/assignment_loss‚3.255725917951116 /loss/FRt4/detection_loss‚2.3525536554780873 .loss/SRqqt1/detection_loss‚2.0933394767344 1loss/SRqqt2/detection_loss‚1.7436008365715252 /loss/FRt4/assignment_loss‚2.815054405819286 2loss/SRqqt1/assignment_loss‚1.3642262036446482 2loss/SRqqt2/assignment_loss‚1.3616326444727533 2loss/SRqqt4/assignment_loss‚1.3320311439911956 0loss/FBt4/assignment_loss‚0.2915819893325014 /loss/FBt1/detection_loss‚2.6448995045253207 /loss/FBt4/detection_loss‚2.0677769354411533 1loss/FBt3/assignment_loss‚0.08314908082213368©&Ûµx’Ë ‚(:‡ ÄûãÔ—Ê gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.292' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘xxÌ ‚(:í ÄûãÔ·¬¤  network.sent‚ 1.9766483e+07 network.recv‚ 2.342258e+06) disk./.usagePercent‚16.61658732617703 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0$ memory_percent‚93.40577125549316" proc.memory.rssMB‚ 2724.7265625 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 303.32421875 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚43826.6357421875 disk.sda.in‚ 0.1953125 disk.rbd1.in‚0" disk.dm-0.out‚43826.6357421875 disk.sda.out‚ 303.32421875 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 303.32421875) proc.memory.percent‚16.63041114807129 disk.sda1.out‚0 disk.rbd1.out‚ 30.3515625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.dm-0.in‚ 1167.7109375' disk.nvme0c0n1.out‚43826.6357421875( proc.memory.availableMB‚ 1080.3984375% disk./.usageGB‚247.59915924072266 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2266.in‚0 cpu‚25.32859989716218*}t’Í ‚(:‡ ÓûãÔÀ‚ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.337' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø ¤©uÎ ‚(:ê ÓûãÔš¦¾  disk.drbd2266.out‚0 disk.drbd2080.out‚0% disk.nvme0n1.out‚43884.2294921875 cpu‚25.327952845236382 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 1167.7109375 network.recv‚ 2.348438e+06 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2265.out‚0) proc.memory.percent‚16.63041114807129% disk./.usageGB‚247.63017654418945 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.rbd1.out‚ 30.3515625 disk.sda.in‚ 0.1953125 disk.dm-1.out‚ 303.734375 network.sent‚ 1.9875204e+07* disk./.usagePercent‚16.618668923437937' disk.nvme0c0n1.out‚43884.2294921875 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 303.734375 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚43884.2294921875 disk.sda3.out‚ 303.734375 disk.sda2.out‚0 disk.dm-0.in‚ 1167.7109375 disk.drbd2265.in‚0$ memory_percent‚93.40755939483643) proc.memory.availableMB‚ 1080.10546875" proc.memory.rssMB‚ 2724.7265625 proc.cpu.threads‚17¯€à’Ï ‚(:‡ âûãÔ×– gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.651' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒûMOÐ ‚(:ô âûãÔ«ì÷& disk.nvme0n1.out‚43952.98876953125 disk.dm-1.out‚ 304.32421875$ disk./.usageGB‚247.6718864440918 disk.drbd1716.in‚0$ memory_percent‚93.40603351593018" proc.memory.rssMB‚ 2724.7265625) proc.memory.percent‚16.63041114807129 disk.rbd1.out‚ 30.4775390625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.sda.out‚ 304.32421875 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚43952.98876953125 disk.nvme0n1.in‚ 1167.7109375# disk.dm-0.out‚43952.98876953125 disk.rbd0.in‚0 disk.dm-0.in‚ 1167.7109375 cpu‚25.327276477048475* disk./.usagePercent‚16.621468109817318 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 304.32421875) proc.memory.availableMB‚ 1080.35546875 network.sent‚ 1.988109e+07 disk.drbd2080.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125 proc.cpu.threads‚17 network.recv‚ 2.352827e+06 disk.drbd2266.out‚0 disk.drbd1716.out‚0¬o<42Ñ - _wandbruntime‚6470  _runtime‚6470íEAîÇÒ Á % _timestamp‚1.7884113676685038e+09 _runtime‚6470.655955642 _step‚682 & lr-AdamW/pg1‚7.272153819470553e-05 & lr-AdamW/pg2‚7.272153819470553e-05 trainer/global_step‚8349:ýþÌÓ Æ & lr-AdamW/pg1‚7.272153819470553e-05 & lr-AdamW/pg2‚7.272153819470553e-05 trainer/global_step‚8349 % _timestamp‚1.7884113676685038e+09 _runtime‚6470.655955642 _step‚682ª0ö—ë2Ô - _wandbruntime‚6470  _runtime‚6470Αü$ç'Õ áO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70667012 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471806214650880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110469197 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484486537753088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157436909 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157436909 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7325240071680 _runtime‚6470.656956381 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278542735 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚529121342070272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491469018224640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121125864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6957451027727 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484486519750144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6983154836992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484486519750144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6983154836992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6982498474496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478763011240851 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121125866 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157416871 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39802185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471806231690240 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121125864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157416871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚529121318130176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562775 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121125866 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚8349 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110469640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536446582141952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491469692590080ZºU ì'Ö æO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39802626 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562775 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121125866 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚8349 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110469640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536446582141952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491469692590080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70667012 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471806214650880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110469197 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484486537753088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157436909 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157436909 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7325240071680 _runtime‚6470.656956381 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278542735 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚529121342070272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491469018224640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121125864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6957451027727 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484486519750144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6983154836992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484486519750144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6983154836992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6982498474496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478763011240851 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121125866 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157416871 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39802185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471806231690240 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121125864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157416871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚529121318130176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121139882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536508902596608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚529182968867840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278595271 L>DeviceStatsMonitor.on_train_batch_end/ê÷«d”allocation.all.allocated‚ 278595271 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478819403283699 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157455389 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491526865070080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39807231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7325933728768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575205 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484542890791936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6957613024935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6983317902336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6983974278144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚529182992807936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚6471.361840559 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121139882 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157455389 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788411368373864e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484542890791936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484542872788992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6983974278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6958267462899 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110481965 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70675180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491526865070080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157435325 _step‚684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536509088050176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121139880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6983317902336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚8349 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70675182 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478819403283699 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157455389 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491526865070080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39807231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7325933728768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575205 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484542890791936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6957613024935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6983317902336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6983974278144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚529182992807936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚6471.361840559 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121139882 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157455389 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788411368373864e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484542890791936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484542872788992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6983974278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6958267462899 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110481965 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70675180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491526865070080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157435325 _step‚684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536509088050176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121139880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6983317902336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚8349 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70675182 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121139882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536508902596608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚529182968867840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278595271 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278595271 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472¬~"Í2Ú - _wandbruntime‚6471  _runtime‚6471^“aî^Û Ø 2loss/SRqqt4/assignment_loss‚1.3723097834384665 1loss/FBt2/assignment_loss‚0.06737366851302795 1loss/FBt3/assignment_loss‚0.11110680226015655 1loss/SRqqt3/detection_loss‚0.9084175229072571 /loss/FBt1/detection_loss‚2.9466063380241394 0loss/FRt2/assignment_loss‚2.9926101830270557 2loss/SRqqt3/assignment_loss‚0.9596735207230918 .loss/FRt4/detection_loss‚2.355902383840361 &loss/total_loss‚42.887746374047225 0loss/FRt1/assignment_loss‚3.1204464764430604 1loss/SRqqt1/assignment_loss‚1.929486917797476 1loss/SRqqt1/detection_loss‚2.3139825286343694 /loss/FBt2/detection_loss‚2.1814877837896347 _step‚685 /loss/FRt3/detection_loss‚1.6044520179275423 0loss/SRqqt2/detection_loss‚1.731397253015767 /loss/FBt3/detection_loss‚0.8044032111155013 _runtime‚6471.363069198 trainer/global_step‚8349 $ _timestamp‚1.788411368375721e+09 /loss/FRt1/detection_loss‚2.9623886983970116 /loss/FRt2/detection_loss‚2.2435337153324517 0loss/FRt3/assignment_loss‚1.2723136370380719 0loss/FBt1/assignment_loss‚0.5044466876424849 2loss/FBt4/assignment_loss‚0.005506466841325164 .loss/FBt4/detection_loss‚4.661641538143158 /loss/FRt4/assignment_loss‚2.648048306620398 2loss/SRqqt2/assignment_loss‚1.3553142405884422 1loss/SRqqt4/detection_loss‚1.8348966919869747 epoch‚6ªîŽÉcÜ Ý trainer/global_step‚8349 $ _timestamp‚1.788411368375721e+09 /loss/FRt1/detection_loss‚2.9623886983970116 /loss/FRt2/detection_loss‚2.2435337153324517 0loss/FRt3/assignment_loss‚1.2723136370380719 0loss/FBt1/assignment_loss‚0.5044466876424849 2loss/FBt4/assignment_loss‚0.005506466841325164 .loss/FBt4/detection_loss‚4.661641538143158 /loss/FRt4/assignment_loss‚2.648048306620398 2loss/SRqqt2/assignment_loss‚1.3553142405884422 1loss/SRqqt4/detection_loss‚1.8348966919869747 epoch‚6 2loss/SRqqt4/assignment_loss‚1.3723097834384665 1loss/FBt2/assignment_loss‚0.06737366851302795 1loss/FBt3/assignment_loss‚0.11110680226015655 1loss/SRqqt3/detection_loss‚0.9084175229072571 /loss/FBt1/detection_loss‚2.9466063380241394 0loss/FRt2/assignment_loss‚2.9926101830270557 2loss/SRqqt3/assignment_loss‚0.9596735207230918 .loss/FRt4/detection_loss‚2.355902383840361 &loss/total_loss‚42.887746374047225 0loss/FRt1/assignment_loss‚3.1204464764430604 1loss/SRqqt1/assignment_loss‚1.929486917797476 1loss/SRqqt1/detection_loss‚2.3139825286343694 /loss/FBt2/detection_loss‚2.1814877837896347 _step‚685 /loss/FRt3/detection_loss‚1.6044520179275423 0loss/SRqqt2/detection_loss‚1.731397253015767 /loss/FBt3/detection_loss‚0.8044032111155013 _runtime‚6471.363069198­¡ RÝ ‚(:’ ñûãÔöÕŠ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.296' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.459199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÆNQvÞ ‚(:ë ñûãÔ¤‰ï  disk.nvme0n1.in‚ 1167.7109375 cpu‚25.326629592643112) disk./.usagePercent‚16.62411445969876% disk./.usageGB‚247.71131896972656 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚44014.646484375 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 304.8671875 disk.sda.in‚ 0.1953125 disk.sda.out‚ 304.8671875 network.sent‚ 1.9887432e+07 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 30.4775390625 disk.dm-0.in‚ 1167.7109375! disk.dm-0.out‚44014.646484375 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 304.8671875" disk.nvme0c0n1.in‚ 1167.7109375& disk.nvme0c0n1.out‚44014.646484375$ memory_percent‚93.40624809265137 disk.sda2.in‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 1080.3203125" proc.memory.rssMB‚ 2724.7265625) proc.memory.percent‚16.63041114807129 disk.rbd0.out‚0 network.recv‚ 2.357291e+06 disk.drbd2265.in‚0 disk.drbd2265.out‚0Ãd[@ß ‚(:’ €üãÔ’‹ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.168' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.233599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Aß.{à ‚(:ð €üãÔÓÙˆ  disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.9995348e+07 disk.sda.in‚ 0.1953125 disk.drbd1716.out‚0 disk.dm-1.out‚ 305.36328125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚44089.419921875" proc.memory.rssMB‚ 2724.7265625 disk.sda3.out‚ 305.36328125" disk.nvme0c0n1.in‚ 1167.7109375 cpu‚25.326029790112774 disk.rbd0.out‚0 disk.nvme0n1.in‚ 1167.7109375$ disk.nvme0n1.out‚44089.419921875 disk.drbd2265.out‚0$ memory_percent‚93.40627193450928* disk./.usagePercent‚16.626153559708985 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0) proc.memory.percent‚16.63041114807129 network.recv‚ 2.362669e+06 disk.drbd1716.in‚0 disk.rbd1.out‚ 30.5400390625 disk.dm-0.in‚ 1167.7109375! disk.dm-0.out‚44089.419921875) proc.memory.availableMB‚ 1080.31640625 proc.cpu.threads‚17% disk./.usageGB‚247.74170303344727 disk.sda.out‚ 305.36328125 disk.drbd2266.in‚0 disk.rbd1.in‚02 á 2á - _wandbruntime‚6507  _runtime‚6507cÇ›&Åâ ¿ & lr-AdamW/pg1‚7.097721224804762e-05 & lr-AdamW/pg2‚7.097721224804762e-05 trainer/global_step‚8399 $ _timestamp‚1.788411404702087e+09 _runtime‚ 6507.68967044 _step‚686ˆƒ+#Êã Ä _step‚686 & lr-AdamW/pg1‚7.097721224804762e-05 & lr-AdamW/pg2‚7.097721224804762e-05 trainer/global_step‚8399 $ _timestamp‚1.788411404702087e+09 _runtime‚ 6507.68967044®®¹JÖ2ä - _wandbruntime‚6507  _runtime‚6507kÈco"å áO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40030659 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121826666 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7368277425664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7023484280320 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚687 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71075412 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121826666 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494327655972864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158362961 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158342923 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280169587 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158342923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487304189695488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7023484280320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚6507.691079719 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539571964620800 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121826664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487304171692544 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111106520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474551438218240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚494328330338304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚487304189695488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7024140642816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494327655972864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481549724585375 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚487304171692544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71075414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474551421178880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111106071 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539572156379136 _MDeviceStatsMonitor.on_train_batch_start/inactive_spliLÝ£óãt_bytes.small_pool.freed‚ 7368109607424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6998286367135 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6997631929379 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚687 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71075412 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121826666 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494327655972864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158362961 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158342923 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280169587 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158342923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487304189695488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7023484280320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚6507.691079719 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539571964620800 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121826664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487304171692544 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111106520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474551438218240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚494328330338304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚487304189695488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7024140642816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494327655972864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481549724585375 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚487304171692544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71075414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474551421178880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111106071 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539572156379136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7368109607424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6998286367135 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6997631929379 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240¯j£óp2ç - _wandbruntime‚6508  _runtime‚6508‘”Öïñ&è ëM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481605445151107 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111118848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚532265529691136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6999102802307 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158381441 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539634485915648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121840682 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121840680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121840680 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487360542734336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121840682 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7024960083968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71083582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40035719 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539634667174912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 % _timestamp‚1.7884114054024236e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚487360524731392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7024303708160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158361377 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494385502818304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474606342348800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280222123 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7024303708160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222123 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7368980164608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494384828439552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7369137483776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚487360542734336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚474606325309440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6998448364343 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202057 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111119301 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8399 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280202057 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494385502818304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532265505751040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158361377 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7024960083968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40035268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚487360524731392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _step‚688 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944ÝÝö&é ðM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222123 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7368980164608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494384828439552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7369137483776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚487360542734336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚474606325309440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6998448364343 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202057 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111119301 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8399 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280202057 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494385502818304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532265505751040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158361377 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7024960083968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40035268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚487360524731392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _step‚688 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481605445151107 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111118848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚532265529691136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6999102802307 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158381441 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539634485915648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121840682 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121840680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121840680 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487360542734336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121840682 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7024960083968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71083582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40035719 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539634667174912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 % _timestamp‚1.7884114054024236e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚487360524731392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7024303708160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158361377 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494385502818304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474606342348800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280222123 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7024303708160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451°èW¿2ê - _wandbruntime‚6508  _runtime‚6508 X)Oµë Ò 1loss/SRqqt2/detection_loss‚1.7762989708118968 /loss/FBt1/detection_loss‚2.8865371147791543 /loss/FBt3/detection_loss‚0.5750427215596216 %loss/total_loss‚41.53276737333393 epoch‚6 _step‚689 1loss/FBt4/assignment_loss‚0.40954378144840575 .loss/FRt3/detection_loss‚1.602515014403678 2loss/SRqqt4/assignment_loss‚1.2032366992475896 1loss/FBt2/assignment_loss‚0.14448476082179695 1loss/SRqqt3/detection_loss‚1.0319528647396765 1loss/SRqqt4/detection_loss‚1.8802604094529762 0loss/FRt3/assignment_loss‚1.4541680419698675 1loss/SRqqt3/assignment_loss‚1.125378329188786 1loss/SRqqt1/detection_loss‚2.5223818229090784 trainer/global_step‚8399 % _timestamp‚1.7884114054043477e+09 /loss/FRt2/assignment_loss‚3.368826898661527 0loss/SRqqt2/assignment_loss‚1.44644893747237 .loss/FRt4/detection_loss‚2.435393760480532 1loss/SRqqt1/assignment_loss‚1.745370145286283 /7‘3£loss/FRt2/detection_loss‚2.0561466054990887 /loss/FBt4/detection_loss‚2.6744529381394386 /loss/FRt1/assignment_loss‚2.951825666016546 1loss/FBt1/assignment_loss‚0.12427366751944646 1loss/FBt3/assignment_loss‚0.23523885803616035 .loss/FBt2/detection_loss‚2.310134246945381 _runtime‚6508.391713096 .loss/FRt4/assignment_loss‚2.70448123681836 /loss/FRt1/detection_loss‚2.8683738811262725¨Ó`Q]ì × /loss/FRt2/assignment_loss‚3.368826898661527 0loss/SRqqt2/assignment_loss‚1.44644893747237 .loss/FRt4/detection_loss‚2.435393760480532 1loss/SRqqt1/assignment_loss‚1.745370145286283 /loss/FRt2/detection_loss‚2.0561466054990887 /loss/FBt4/detection_loss‚2.6744529381394386 /loss/FRt1/assignment_loss‚2.951825666016546 1loss/FBt1/assignment_loss‚0.12427366751944646 1loss/FBt3/assignment_loss‚0.23523885803616035 .loss/FBt2/detection_loss‚2.310134246945381 _runtime‚6508.391713096 .loss/FRt4/assignment_loss‚2.70448123681836 /loss/FRt1/detection_loss‚2.8683738811262725 1loss/SRqqt2/detection_loss‚1.7762989708118968 /loss/FBt1/detection_loss‚2.8865371147791543 /loss/FBt3/detection_loss‚0.5750427215596216 %loss/total_loss‚41.53276737333393 epoch‚6 _step‚689 1loss/FBt4/assignment_loss‚0.40954378144840575 .loss/FRt3/detection_loss‚1.602515014403678 2loss/SRqqt4/assignment_loss‚1.2032366992475896 1loss/FBt2/assignment_loss‚0.14448476082179695 1loss/SRqqt3/detection_loss‚1.0319528647396765 1loss/SRqqt4/detection_loss‚1.8802604094529762 0loss/FRt3/assignment_loss‚1.4541680419698675 1loss/SRqqt3/assignment_loss‚1.125378329188786 1loss/SRqqt1/detection_loss‚2.5223818229090784 trainer/global_step‚8399 % _timestamp‚1.7884114054043477e+09±cV$í ‚(:’ üãÔé‡ gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.631' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.126200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o¼ƒùsî ‚(:è üãÔñ  network.sent‚ 2.0001422e+07 disk.drbd1716.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.in‚0 disk.sda3.out‚ 305.73828125& disk.nvme0c0n1.out‚44152.173828125( proc.memory.percent‚16.6304349899292$ disk.nvme0n1.out‚44152.173828125! disk.dm-0.out‚44152.173828125 disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0 disk.sda1.out‚0 cpu‚25.32540398042109 network.recv‚ 2.366871e+06 disk.dm-0.in‚ 1167.7109375 disk.dm-1.out‚ 305.73828125 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2724.73046875 disk.rbd1.in‚0 disk.rbd1.out‚ 30.5400390625 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.sda.out‚ 305.73828125$ memory_percent‚93.40667724609375$ disk./.usageGB‚247.7501449584961 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.availableMB‚1080.25 proc.cpu.threads‚17* disk./.usagePercent‚16.626720104382258 disk.drbd2266.in‚0 disk.drbd2266.out‚0ÅKžï ‚(:’ žüãÔÒ³† gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.296' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.459199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 QfÕið ‚(:Þ žüãÔÙ‘  disk.dm-0.in‚ 1167.7109375 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2724.73046875# disk.nvme0c0n1.out‚ 44226.359375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd1.in‚0 disk.dm-1.out‚ 306.515625 disk.sda.out‚ 306.515625 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17 network.sent‚ 2.0109938e+07 disk.rbd1.out‚ 30.666015625 disk.nvme0n1.in‚ 1167.7109375 disk.sda3.out‚ 306.515625$ memory_percent‚93.40674877166748 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.out‚ 44226.359375$ disk./.usageGB‚247.7965431213379 disk.drbd1716.out‚0! disk.nvme0n1.out‚ 44226.359375) proc.memory.availableMB‚ 1080.23828125( proc.memory.percent‚16.6304349899292* disk./.usagePercent‚16.629833924021224 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 cpu‚25.32473485203789 network.recv‚ 2.373079e+06 disk.drbd2080.out‚0 disk.rbd0.out‚0æ[±F’ñ ‚(:‡ ­üãÔ®Åð gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ~|ò ‚(:ñ ­üãÔ¤‰ã  network.sent‚ 2.0116286e+07 disk.drbd2265.out‚0' disk.nvme0c0n1.out‚44300.4521484375" disk.dm-0.out‚44300.4521484375) proc.memory.availableMB‚ 1080.23828125 proc.cpu.threads‚17 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2724.73046875* disk./.usagePercent‚16.631128554121876% disk./.usageGB‚247.81583404541016 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 1167.7109375% disk.nvme0n1.out‚44300.4521484375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 306.82421875 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 306.82421875 cpu‚25.32406397830621 network.recv‚ 2.377381e+06 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.sda.out‚ 306.82421875 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 30.666015625 disk.nvme0n1.in‚ 1167.7109375$ memory_percent‚93.40674877166748( proc.memory.percent‚16.6304349899292çyª2ó - _wandbruntime‚6544  _runtime‚6544íyNÁÇô Á % _timestamp‚1.7884114413147902e+09 _runtime‚6544.302231356 _step‚690 & lr-AdamW/pg1‚6.924675570359719e-05 & lr-AdamW/pg2‚6.924675570359719e-05 trainer/global_step‚8449‡r4¡Ìõ Æ & lr-AdamW/pg1‚6.924675570359719e-05 & lr-AdamW/pg2‚6.924675570359719e-05 trainer/global_step‚8449 % _timestamp‚1.7884114413147902e+09 _runtime‚6544.302231356 _step‚690²{àË2ö - _wandbruntime‚6544  _runtime‚6544{’òæ'÷ àO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490121841637888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497186968086528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122527466 trainer/global_step‚8449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71483814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71483812 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7411020113920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚477296644746240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚477296627706880 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122527464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281816479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788411441315501e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497186293721088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159289013 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159268975 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122527466 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122527464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542697412010496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484335766452783 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40259420 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7065126448640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535286391896576 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 _step‚691 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159289013 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542697607963136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111743677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7411192126464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7039121706543 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7038467268787 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281796439 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7064470086144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚535286415836672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497186293721088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7065126448640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159268975 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40259863 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _runtime‚6544.303147695 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111743232€Ú‰_ë'ø åO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚477296644746240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚477296627706880 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122527464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281816479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788411441315501e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497186293721088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159289013 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159268975 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122527466 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122527464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542697412010496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484335766452783 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40259420 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7065126448640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535286391896576 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 _step‚691 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159289013 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542697607963136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111743677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7411192126464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7039121706543 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7038467268787 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281796439 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7064470086144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚535286415836672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497186293721088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7065126448640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159268975 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40259863 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _runtime‚6544.303147695 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111743232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490121841637888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497186968086528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122527466 trainer/global_step‚8449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71483814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71483812 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7411020113920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0³žLŒp2ù - _wandbruntime‚6544  _runtime‚6544—ÀYVrú ëM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8449 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159287429 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159287429 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535348066574336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111755892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚477351531837440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490178176673792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7065945889792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚6544.989231434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111756354 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7065945889792 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848975 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7039283703751 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122541480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542759910435328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚477351548876800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490178176673792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7065289513984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.'©B]allocated‚ 159307493 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122541480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40264372 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159307493 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40263912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7039938141715 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71491982 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884114420011854e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281848975 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490178194676736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122541482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884114420011854e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281848975 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490178194676736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122541482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8449 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159287429 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159287429 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535348066574336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111755892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚477351531837440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490178176673792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7065945889792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚6544.989231434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111756354 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7065945889792 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848975 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7039283703751 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122541480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542759910435328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚477351548876800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490178176673792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7065289513984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159307493 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122541480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40264372 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159307493 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40263912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414´Õ'¯2ü - _wandbruntime‚6544  _runtime‚6544ºŠhSXý Ò 1loss/FBt3/assignment_loss‚0.12629183300517005 .loss/FRt2/detection_loss‚1.967440776122167 1loss/FBt1/assignment_loss‚0.09589777622386464 0loss/SRqqt1/detection_loss‚2.511533348791061 0loss/SRqqt4/detection_loss‚1.796528710535279 /loss/FBt4/detection_loss‚2.1852771162986757 _runtime‚6544.990587302 epoch‚6 /loss/FRt4/detection_loss‚2.4917510416772632 /loss/FBt1/detection_loss‚2.6051595211029053 .loss/FBt2/detection_loss‚2.115303558962686 /loss/FRt1/assignment_loss‚3.293545945694572 0loss/FRt4/assignment_loss‚3.2164519561661615 1loss/SRqqt1/assignment_loss‚1.698435973976889 1loss/SRqqt2/assignment_loss‚1.391398040461354 _step‚693 1loss/SRqqt2/detection_loss‚1.6460926316212863 %loss/total_loss‚41.82350404207438 .loss/FRt3/assignment_loss‚1.76304606454713 2loss/SRqqt4/assignment_loss‚1.4629352902217458 % _timestamp‚1.7884114420032458e+09 2loss/SRqqt3/assignment_loss‚0.8699849422923044 0loss/FBt2/assignment_loss‚0.2640136691292615 1loss/SRqqt3/detection_loss‚1.1667572151412045 /loss/FBt3/detection_loss‚0.5743371332368539 trainer/global_step‚8449 0loss/FBt4/assignment_loss‚0.5694781193509698 /loss/FRt1/detection_loss‚3.5492479989403174 0loss/FRt2/assignment_loss‚2.6734564326026224 /loss/FRt3/detection_loss‚1.7891389459726357…†Áþ]þ × 2loss/SRqqt3/assignment_loss‚0.8699849422923044 0loss/FBt2/assignment_loss‚0.2640136691292615 1loss/SRqqt3/detection_loss‚1.1667572151412045 /loss/FBt3/detection_loss‚0.5743371332368539 trainer/global_step‚8449 0loss/FBt4/assignment_loss‚0.5694781193509698 /loss/FRt1/detection_loss‚3.5492479989403174 0loss/FRt2/assignment_loss‚2.6734564326026224 /loss/FRt3/detection_loss‚1.7891389459726357 1loss/FBt3/assignment_loss‚0.12629183300517005 .loss/FRt2/detection_loss‚1.967440776122167 1loss/FBt1/assignment_loss‚0.09589777622386464 0loss/SRqqt1/detection_loss‚2.511533348791061 0loss/SRqqt4/detection_loss‚1.796528710535279 /loss/FBt4/detection_loss‚2.1852771162986757 _runtime‚6544.990587302 epoch‚6 /loss/FRt4/detection_loss‚2.4917510416772632 /loss/FBt1/detection_loss‚2.6051595211029053 .loss/FBt2/detection_loss‚2.115303558962686 /loss/FRt1/assignment_loss‚3.293545945694572 0loss/FRt4/assignment_loss‚3.2164519561661615 1loss/SRqqt1/assignment_loss‚1.698435973976889 1loss/SRqqt2/assignment_loss‚1.391398040461354 _step‚693 1loss/SRqqt2/detection_loss‚1.6460926316212863 %loss/total_loss‚41.82350404207438 .loss/FRt3/assignment_loss‚1.76304606454713 2loss/SRqqt4/assignment_loss‚1.4629352902217458 % _timestamp‚1.7884114420032458e+09µ·•™‘’ÿ ‚(:‡ ¼üãÔÖ…Ð  gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.152' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V4Vµ€!‚(:ô ¼üãÔ¼¶œ disk.drbd2266.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚44357.36083984375 disk.rbd1.in‚0& disk.nvme0n1.out‚44357.36083984375 disk.drbd2266.out‚0 disk.drbd2080.in‚0) proc.memory.availableMB‚ 1002.95703125* proc.memory.percent‚17.101144790649414 disk.nvme0n1.in‚ 1167.7109375 disk.dm-1.out‚ 307.48828125 disk.sda.in‚ 0.1953125 disk.sda1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2801.8515625 disk.drbd2265.in‚0 disk.sda.out‚ 307.48828125 cpu‚25.32393519516259 proc.cpu.threads‚17 network.recv‚ 2.381103e+06 disk.dm-0.in‚ 1167.7109375 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 307.48828125 disk.drbd2080.out‚0 disk.drbd1716.in‚0# disk.dm-0.out‚44357.36083984375 disk.dm-1.in‚ 0.1953125 disk.sda1.out‚0 network.sent‚ 2.0134597e+07 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 1167.7109375% disk./.usageGB‚247.84227752685547 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚93.87843608856201) disk./.usagePercent‚16.63290319834923 disk.rbd1.out‚ 30.728515625 disk.drbd2265.out‚0 Åhw‘!‚(:† ËüãÔ‚¬Š gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.322' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü 8|‚!‚(:ñ ËüãÔþ·í network.recv‚ 2.38718e+06* disk./.usagePercent‚16.633577266186887 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 30.814453125 disk.dm-0.in‚ 1167.7109375# disk.dm-0.out‚44427.08740234375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.1953125) proc.memory.availableMB‚ 1002.30859375 network.sent‚ 2.0230509e+07 disk.sda1.out‚0 disk.dm-1.out‚ 307.8984375% disk./.usageGB‚247.85232162475586 disk.sda2.out‚0( disk.nvme0c0n1.out‚44427.08740234375 disk.sda3.out‚ 307.8984375 disk.drbd2080.in‚0* proc.memory.percent‚17.101144790649414" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 1167.7109375& disk.nvme0n1.out‚44427.08740234375 disk.sda.out‚ 307.8984375 disk.drbd2266.out‚0 disk.rbd0.in‚0# memory_percent‚93.8823938369751 disk.sda1.in‚0 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2801.8515625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0 cpu‚25.334310524997885 ‹Ä’ƒ!‚(:‡ ÚüãÔýÜô gpu.0.gpu‚14.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—XM{„!‚(:ð ÚüãÔ¯Æ # disk.dm-0.out‚44485.81494140625 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 308.5859375* proc.memory.percent‚17.156553268432617 network.sent‚ 2.0236443e+07 disk.sda3.in‚ 0.1953125 disk.sda1.out‚0( proc.memory.availableMB‚ 993.46484375 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚44485.81494140625 disk.dm-0.in‚ 1167.7109375 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚16.63447278054166 disk.rbd1.out‚ 30.970703125$ memory_percent‚93.93637180328369 proc.cpu.threads‚17 disk.sda3.out‚ 308.5859375" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚44485.81494140625% disk./.usageGB‚247.86566543579102 disk.sda.in‚ 0.1953125 disk.nvme0n1.in‚ 1167.7109375" proc.memory.rssMB‚ 2810.9296875 cpu‚25.34420744760666 network.recv‚ 2.391393e+06 disk.drbd2265.out‚0 disk.sda.out‚ 308.5859375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0-ã„’…!‚(:‡ éüãÔš‡˜ gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+Â{Á}†!‚(:ò éüãÔõèš % disk./.usageGB‚247.88129806518555 disk.drbd2266.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2810.9296875 disk.sda3.in‚ 0.1953125 network.recv‚ 2.395977e+06 disk.dm-0.in‚ 1167.7109375 disk.rbd0.in‚0 disk.rbd1.in‚0 network.sent‚ 2.0242793e+07 disk.sda3.out‚ 309.0390625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1167.7109375( disk.nvme0c0n1.out‚44556.38427734375 disk.sda.out‚ 309.0390625 disk.drbd2265.in‚0( proc.memory.availableMB‚ 992.95703125 cpu‚25.354718247965582* disk./.usagePercent‚16.635521899416975 disk.rbd1.out‚ 30.970703125 disk.dm-1.out‚ 309.0390625 disk.drbd1716.in‚0$ memory_percent‚93.93947124481201 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚44556.38427734375* proc.memory.percent‚17.156553268432617# disk.dm-0.out‚44556.38427734375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125" disk.nvme0c0n1.in‚ 1167.7109375ÐÞ$2‡!- _wandbruntime‚6602  _runtime‚6602éVkÊÏ"ˆ!ÈT _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 283141907 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112460158 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚154 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 159888725 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 283157886 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚545446748179968 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚8462 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492715455592960 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 159904702 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚71916862 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚492715437590016 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 283141907 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚538005437105664 _runtime‚6602.327015143 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚40543296 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40543142 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 283157886 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚156 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7067894840299 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚492715455592960 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚694 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 123253182 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 123253184 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚499809960998400 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚499809450165248 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMoniÒWQ6tor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚499809960998400 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚499809450165248 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7094505405440 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚492715437590016 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 159888725 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112460158 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚154 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 159888725 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 283157886 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚545446748179968 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚8462 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492715455592960 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 159904702 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚71916862 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚492715437590016 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 283141907 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚538005437105664 _runtime‚6602.327015143 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚40543296 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40543142 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 283157886 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚156 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7067894840299 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚492715455592960 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚694 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 123253182 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 123253184 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚499809960998400 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚499809450165248 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚499809960998400 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚499809450165248 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7094505405440 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚492715437590016 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 159888725 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 159912004 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7441565051392 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112471146 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚6603.238397365 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112470990 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚71923364 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7094870532096 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚492752285621760 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 123263990 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 135089664 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 7068749364099 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚545484989040128 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚154 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚492752267618816 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 159896027 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40547626 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚71923366 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0Ѭø²$Œ!×R _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚479829973509120 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚499847156153856 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7094377701888 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚545484853950464 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 123263990 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7094870532096 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 283175994 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚486898231958827 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚499847156153856 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111149568 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚486898739912579 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 283175994 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚499846645320704 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7094377701888 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚538043288899072 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 7068258449707 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚492752285621760 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚492752267618816 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 159896027 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 283160015 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚71923366 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 283160015 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 123263988 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚156 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚538043312839168 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚40547780 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 123263988 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7441676200960 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚499846645320704 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 159912004 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 159912004 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7441565051392 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112471146 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚6603.238397365 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112470990 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚71923364 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7094870532096 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚492752285621760 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 123263990 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 135089664 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 7068749364099 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚545484989040128 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚4"›f Ü30 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚154 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚492752267618816 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 159896027 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40547626 B&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918437600135803 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17426900565624237 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_6_of_7‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 jet/accuracy_3_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35299721360206604 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27719423174858093 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002543752547353506 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19311173260211945 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27729883790016174 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15298375487327576 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5064436197280884 particle/accuracy_1_of_5‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4794315993785858 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18673083186149597 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_1_of_2‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05692048370838165 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22966112196445465 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06028369069099426 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0632183849811554 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6176281571388245 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4577508270740509 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0800417959690094 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15248480439186096 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2082769125699997 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9882669448852539 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928812980651855 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4668251574039459 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17667926847934723 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_4‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166567325592 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.17307692766189575 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 particle/accuracy_7_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.435585618019104 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3847442865371704 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12802618741989136 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.519909679889679 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4927448034286499 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38124993443489075 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8572772741317749 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 )jet/accuracy_1_of_2‚0.737786054611206 particle/accuracy_3_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013240405358374119 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.023930305615067482 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 -jet/accuracy_5_of_5‚0.0026881720405071974 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16075526177883148 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2341269999742508 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5446428656578064 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6874999403953552 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4399225115776062 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2340909093618393 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 _step‚696 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15436452627182007 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2693902850151062 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018564797937870026 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35902175307273865 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.703498363494873 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4563186764717102 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515305042267 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4950437843799591 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4722559154033661 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.991738498210907 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1769389808177948 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,jet/accuracy_3_of_3‚0.026628078892827034 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19509032368659973 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5095145106315613 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17032864689826965 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2527027130126953 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 )jet/accuracy_2_of_5‚0.448207825422287 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39249247312545776 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10536399483680725 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07765419781208038 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2963336110115051 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5268563628196716 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902167916297913 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1739591509103775 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06469038128852844 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5646470785140991 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.371296226978302 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2121913731098175 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5058704018592834 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29862740635871887 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6261126399040222 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3776688277721405 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5125904679298401 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4858199954032898 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4273946285247803 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11665009707212448 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963507652282715 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5177223086357117 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5363275408744812 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11592422425746918 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.37800031900405884 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01744186133146286 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9828309416770935 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5422427654266357 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911775588989258 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5067957043647766 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15111111104488373 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47999998927116394 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10476190596818924 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46983110904693604 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41301628947257996 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16942386329174042 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988226294517517 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01031745970249176 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5189864635467529 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03499434515833855 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1541232168674469 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3858630955219269 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37803560495376587 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017965367063879967 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9412250518798828 particle/accuracy_1_of_3‚1 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4722559154033661 ,jet/accuracy_4_of_4‚0.006960703991353512 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4651750326156616 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921432733535767 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.276777446269989 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6516903638839722 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.210664764046669 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17607712745666504 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2101432979106903 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2631153464317322 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 particle/accuracy_3_of_6‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915125966072083 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_6_of_6‚1 jet/accuracy_7_of_7‚0 particle/accuracy_4_of_7‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02358490601181984 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1769389808177948 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.07692307978868484 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 trainer/global_step‚8462 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979721903800964 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3937886953353882 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47736671566963196 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3713342845439911 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3776664137840271 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.41498783230781555 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 % _timestamp‚1.7884115129343088e+09 )loss/val_total_loss‚41.83433840400233 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013256724923849106 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9919999241828918 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10301528871059418 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22678637504577637 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2067171335220337 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4359094202518463 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14989908039569855 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,particle/sensitivity‚0.32868683338165283 &particle/f_score‚0.412259578704834 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4417683482170105 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30877745151519775 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5463285446166992 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4258284568786621 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.38856208324432373 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_4_of_5‚0.02060931921005249 particle/accuracy_2_of_7‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16501791775226593 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28333333134651184 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2765424847602844 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5256046056747437 jet/accuracy_1_of_7‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8234986662864685 jet/accuracy_4_of_7‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920984506607056 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 particle/accuracy_4_of_6‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021919880528002977 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4452040195465088 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.37104374170303345 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5469135642051697 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.277777761220932 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 jet/accuracy_2_of_7‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909665584564209 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7"Purity/*FRt*SRqqt*FBt/event_purity‚0.24837626516819 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5507240295410156 7 Purity/0FRt3SŸÁT;ÐRqqt*FBt/FBt_purity‚0.8888888955116272 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984404444694519 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.509202241897583 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11747529357671738 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4642096757888794 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5627976059913635 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8537790775299072 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.277777761220932 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.47039473056793213 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0793650820851326 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9972789287567139 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1851678341627121 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 epoch‚6 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45378294587135315 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 5validation_average_jet_accuracy‚0.430155873298645 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2258506864309311 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5556213855743408 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903312921524048 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00522132171317935 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2906431555747986 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15226046741008759 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5931758284568787 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02718784473836422 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4609416425228119 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.25106126070022583 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 _runtime‚6615.923936717 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5281133651733398 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_6‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5696015357971191 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14634670317173004 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4583333432674408 *jet/accuracy_2_of_6‚0.5705128312110901 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4760441780090332 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0381944440305233 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5488505959510803 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5821748375892639 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.019736841320991516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_5_of_5‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16668932139873505 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17005856335163116 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2206021249294281 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24803254008293152 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17265254259109497 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5875272154808044 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6135416030883789 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4521551728248596 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9890003800392151 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.4736120402812958 jet/accuracy_6_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19051723182201385 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03817712515592575 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5339506268501282 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303į"a‘!¡§ , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 trainer/global_step‚8462 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979721903800964 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3937886953353882 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47736671566963196 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3713342845439911 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3776664137840271 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.41498783230781555 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 % _timestamp‚1.7884115129343088e+09 )loss/val_total_loss‚41.83433840400233 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013256724923849106 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9919999241828918 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10301528871059418 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22678637504577637 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2067171335220337 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4359094202518463 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14989908039569855 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,particle/sensitivity‚0.32868683338165283 &particle/f_score‚0.412259578704834 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4417683482170105 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30877745151519775 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5463285446166992 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4258284568786621 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.38856208324432373 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_4_of_5‚0.02060931921005249 particle/accuracy_2_of_7‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16501791775226593 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28333333134651184 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2765424847602844 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5256046056747437 jet/accuracy_1_of_7‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8234986662864685 jet/accuracy_4_of_7‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920984506607056 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 particle/accuracy_4_of_6‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021919880528002977 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4452040195465088 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.37104374170303345 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5469135642051697 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.277777761220932 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 jet/accuracy_2_of_7‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909665584564209 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7"Purity/*FRt*SRqqt*FBt/event_purity‚0.24837626516819 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5507240295410156 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984404444694519 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.509202241897583 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11747529357671738 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4642096757888794 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5627976059913635 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8537790775299072 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.277777761220932 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.47039473056793213 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0793650820851326 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9972789287567139 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1851678341627121 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 epoch‚6 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45378294587135315 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 5validation_average_jet_accuracy‚0.430155873298645 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2258506864309311 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5556213855743408 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903312921524048 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00522132171317935 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2906431555747986 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15226046741008759 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5931758284568787 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02718784473836422 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4609416425228119 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.25106126070022583 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 _runtime‚6615.923936717 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5281133651733398 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_6‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5696015357971191 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14634670317173004 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4583333432674408 *jet/accuracy_2_of_6‚0.5705128312110901 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4760441780090332 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0381944440305233 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5488505959510803 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5821748375892639 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.019736841320991516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_5_of_5‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16668932139873505 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17005856335163116 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2206021249294281 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24803254008293152 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17265254259109497 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5875272154808044 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6135416030883789 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4521551728248596 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9890003800392151 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.4736120402812958 jet/accuracy_6_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19051723182201385 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03817712515592575 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5339506268501282 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48565033078193665 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.038702212274074554 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.040360983461141586 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_4_of_4‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17643681168556213 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4220033586025238 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39604896306991577 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5815796256065369 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18305639922618866 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 *jet/accuracy_1_of_4‚0.9303489923477173 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4142112731933594 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4500785768032074 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *particle/specificity‚0.950947642326355 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5208333134651184 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4259259104728699 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6874999403953552 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11621436476707458 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918437600135803 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17426900565624237 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_6_of_7‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 jet/accuracy_3_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35299721360206604 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27719423174858093 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002543752547353506 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19311173260211945 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27729883790016174 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15298375487327576 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5064436197280884 particle/accuracy_1_of_5‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4794315993785858 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18673083186149597 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_1_of_2‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05692048370838165 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22966112196445465 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06028369069099426 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0632183849811554 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6176281571388245 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4577508270740509 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0800417959690094 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15248480439186096 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2082769125699997 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9882669448852539 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928812980651855 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4668251574039459 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17667926847934723 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_4‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166567325592 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.17307692766189575 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 particle/accuracy_7_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.435585618019104 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3847442865371704 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12802618741989136 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.519909679889679 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4927448034286499 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38124993443489075 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8572772741317749 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 )jet/accuracy_1_of_2‚0.737786054611206 particle/accuracy_3_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013240405358374119 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.023930305615067482 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 -jet/accuracy_5_of_5‚0.0026881720405071974 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16075526177883148 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2341269999742508 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5446428656578064 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt’b†2/SRqqt_purity‚0.6874999403953552 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4399225115776062 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2340909093618393 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 _step‚696 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15436452627182007 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2693902850151062 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018564797937870026 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35902175307273865 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.703498363494873 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4563186764717102 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515305042267 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4950437843799591 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4722559154033661 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.991738498210907 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1769389808177948 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,jet/accuracy_3_of_3‚0.026628078892827034 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19509032368659973 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5095145106315613 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17032864689826965 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2527027130126953 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 )jet/accuracy_2_of_5‚0.448207825422287 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39249247312545776 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10536399483680725 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07765419781208038 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2963336110115051 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5268563628196716 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902167916297913 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1739591509103775 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06469038128852844 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5646470785140991 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.371296226978302 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2121913731098175 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5058704018592834 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29862740635871887 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6261126399040222 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3776688277721405 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5125904679298401 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4858199954032898 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4273946285247803 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11665009707212448 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963507652282715 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5177223086357117 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5363275408744812 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11592422425746918 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.37800031900405884 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01744186133146286 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9828309416770935 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5422427654266357 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911775588989258 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5067957043647766 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15111111104488373 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47999998927116394 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10476190596818924 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46983110904693604 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41301628947257996 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16942386329174042 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988226294517517 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01031745970249176 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5189864635467529 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03499434515833855 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1541232168674469 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3858630955219269 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37803560495376587 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017965367063879967 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9412250518798828 particle/accuracy_1_of_3‚1 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4722559154033661 ,jet/accuracy_4_of_4‚0.006960703991353512 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4651750326156616 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921432733535767 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.276777446269989 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6516903638839722 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.210664764046669 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17607712745666504 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2101432979106903 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2631153464317322 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 particle/accuracy_3_of_6‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915125966072083 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_6_of_6‚1 jet/accuracy_7_of_7‚0 particle/accuracy_4_of_7‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02358490601181984 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1769389808177948 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.07692307978868484¸Lw;Ï’!‚(:’ ‡ýãÔœüî gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.631' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.126200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{0 ·|“!‚(:ñ ‡ýãÔÊÿì  disk.sda3.out‚ 310.05859375 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0% disk.nvme0n1.out‚45828.3271484375 disk.dm-1.out‚ 310.05859375 disk.drbd1716.in‚0$ memory_percent‚91.30597114562988 disk.sda.out‚ 310.05859375 disk.drbd2080.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1424.4296875* proc.memory.percent‚17.162561416625977 disk.drbd2266.out‚0 network.sent‚ 2.0356575e+07" disk.dm-0.out‚45828.3271484375" disk.nvme0c0n1.in‚ 1167.7109375 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0) disk./.usagePercent‚16.63785873618952 disk.dm-0.in‚ 1167.7109375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125" proc.memory.rssMB‚ 2811.9140625 disk.rbd1.out‚ 31.033203125 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.363895569461945 network.recv‚ 2.405616e+06% disk./.usageGB‚247.91611862182617 disk.sda2.in‚0' disk.nvme0c0n1.out‚45828.3271484375 disk.drbd1716.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17Œá$Lœ”!‚(:‘ –ýãÔüûü gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.193' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.83860000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ$ûw•!‚(:ì –ýãÔ”¼´  disk.drbd2265.out‚0 disk.sda3.out‚ 310.734375 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 310.734375" disk.nvme0c0n1.in‚ 1167.7109375( disk.nvme0c0n1.out‚45906.87353515625 disk.rbd1.out‚ 31.1591796875 disk.drbd2080.out‚0 network.recv‚ 2.412078e+06$ disk./.usageGB‚247.9445037841797 disk.sda3.in‚ 0.1953125 cpu‚25.363185294693167 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 1167.7109375$ memory_percent‚91.30449295043945 disk.sda2.out‚0 disk.dm-0.in‚ 1167.7109375# disk.dm-0.out‚45906.87353515625& disk.nvme0n1.out‚45906.87353515625( disk./.usagePercent‚16.6397636882518 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.sent‚ 2.0480679e+07 disk.sda.out‚ 310.734375 disk.drbd2266.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1424.671875" proc.memory.rssMB‚ 2811.9140625 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.162561416625977’Í, 2–!- _wandbruntime‚6646  _runtime‚6646yÛ~¿Æ—!À _runtime‚ 6646.80240505 _step‚697 & lr-AdamW/pg1‚6.753052918590446e-05 & lr-AdamW/pg2‚6.753052918590446e-05 trainer/global_step‚8499 % _timestamp‚1.7884115438149076e+09¼Ÿí˘!Å & lr-AdamW/pg1‚6.753052918590446e-05 & lr-AdamW/pg2‚6.753052918590446e-05 trainer/global_step‚8499 % _timestamp‚1.7884115438149076e+09 _runtime‚ 6646.80240505 _step‚697¹wa¡·2™!- _wandbruntime‚6646  _runtime‚6646•Á›Ãç'š!áO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123909044 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586072 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548232632486400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540755099990528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160677028 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495259767391744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123909044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160656990 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502388243253760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚548232428145152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7129132224512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495259785394688 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113069819 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7102227108277 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚495259767391744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180401152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502388243253760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7477508555776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚698 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7128475862016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123909042 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482265358121984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123909042 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160656990 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160677028 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72301964 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540755123930624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7102881546033 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40768310 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8499 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502388917619200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7128475862016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113070276 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40767855 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204341248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495259785394688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284566032 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502388917619200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6646.803542285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72301966 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482265341082624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489367568190901 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7129132224512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚4141n¢%›!æO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚698 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7128475862016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123909042 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482265358121984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123909042 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160656990 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160677028 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72301964 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540755123930624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7102881546033 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40768310 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8499 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502388917619200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7128475862016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113070276 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40767855 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204341248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495259785394688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284566032 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502388917619200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6646.803542285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72301966 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482265341082624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489367568190901 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7129132224512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123909044 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586072 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548232632486400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540755099990528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160677028 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495259767391744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123909044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160656990 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502388243253760 OIDeviceStatsMonitor.on_train_batch_startŦ Ç/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚548232428145152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7129132224512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495259785394688 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113069819 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7102227108277 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚495259767391744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180401152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502388243253760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7477508555776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0º;ójz2œ!- _wandbruntime‚6647  _runtime‚6647ßCïâñ&!ëM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129951665664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502445415720448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚495316138433536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚699 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129951665664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160675444 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6647.502260646 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548295127793152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482320262252544 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚495316120430592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123923060 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160695508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7478181125632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123923058 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540816774668288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113082462 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548294931853824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚495316120430592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502446090099200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113082911 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489423288756425 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884115445143054e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284598502 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚8499 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123923058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40772777 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160675444 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7129295289856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7103697981205 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284618568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540816750728192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72310134 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489423960233749 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123923058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40772777 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160675444 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7129295289856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7103697981205 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284618568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540816750728192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72310134 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489423960233749 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129951665664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502445415720448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚495316138433536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚699 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129951665664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160675444 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6647.502260646 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548295127793152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482320262252544 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚495316120430592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123923060 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160695508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7478181125632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123923058 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540816774668288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113082462 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548294931853824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚495316120430592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502446090099200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113082911 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489423288756425 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884115445143054e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284598502 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚8499»O„ó02Ÿ!- _wandbruntime‚6647  _runtime‚6647f,rPZ !Ô 1loss/FBt4/assignment_loss‚0.13887192722177133 0loss/FRt1/assignment_loss‚3.1738868040197037 0loss/FRt2/assignment_loss‚3.1309254182421644 _step‚700 1loss/SRqqt3/detection_loss‚0.8225709558946306 -loss/FBt1/detection_loss‚2.45488235780171 /loss/FBt4/detection_loss‚2.4014383032917976 %loss/total_loss‚41.34529695281486 trainer/global_step‚8499 0loss/FRt4/assignment_loss‚2.5086578826109567 0loss/SRqqt1/detection_loss‚2.524059796774829 /loss/FBt3/detection_loss‚0.5962461179334914 2loss/SRqqt4/assignment_loss‚1.5019288935205515 0loss/FBt2/assignment_loss‚0.3701045716491838 .loss/FRt2/detection_loss‚2.139260772220816 /loss/FRt3/detection_loss‚1.6001326134749527 /loss/FRt4/detection_loss‚2.1010602444410322 0loss/SRqqt4/detection_loss‚1.774528392974068 % _timestamp‚1.7884115445158343e+09 0loss/FRt3/assignment_loss‚1.6300087047034297 _runtime‚6647.503204476 2loss/SRqqt1/assignment_loss‚1.4590492079517356 0loss/FBt3/assignment_loss‚0.1384237708679497 .loss/FRt1/detection_loss‚2.886862379663131 .loss/FBt2/detection_loss‚4.065021753311157 epoch‚7 2loss/SRqqt2/assignment_loss‚1.3417123318028947 2loss/SRqqt3/assignment_loss‚0.9555625231647543 1loss/FBt1/assignment_loss‚0.11972848018298723 1loss/SRqqt2/detection_loss‚1.5103727490951617Dgr_¡!Ù .loss/FRt2/detection_loss‚2.139260772220816 /loss/FRt3/detection_loss‚1.6001326134749527 /loss/FRt4/detection_loss‚2.1010602444410322 0loss/SRqqt4/detection_loss‚1.774528392974068 % _timestamp‚1.7884115445158343e+09 0loss/FRt3/assignment_loss‚1.6300087047034297 _runtime‚6647.503204476 2loss/SRqqt1/assignment_loss‚1.4590492079517356 0loss/FBt3/assignment_loss‚0.1384237708679497 .loss/FRt1/detection_loss‚2.886862379663131 .loss/FBt2/detection_loss‚4.065021753311157 epoch‚7 2loss/SRqqt2/assignment_loss‚1.3417123318028947 2loss/SRqqt3/assignment_loss‚0.9555625231647543 1loss/FBt1/assignment_loss‚0.11972848018298723 1loss/SRqqt2/detection_loss‚1.5103727490951617 1loss/FBt4/assignment_loss‚0.13887192722177133 0loss/FRt1/assignment_loss‚3.1738868040197037 0loss/FRt2/assignment_loss‚3.1309254182421644 _step‚700 1loss/SRqqt3/detection_loss‚0.8225709558946306 -loss/FBt1/detection_loss‚2.45488235780171 /loss/FBt4/detection_loss‚2.4014383032917976 %loss/total_loss‚41.34529695281486 trainer/global_step‚8499 0loss/FRt4/assignment_loss‚2.5086578826109567 0loss/SRqqt1/detection_loss‚2.524059796774829 /loss/FBt3/detection_loss‚0.5962461179334914 2loss/SRqqt4/assignment_loss‚1.5019288935205515 0loss/FBt2/assignment_loss‚0.3701045716491838¼Ø?¶’¢!‚(:‡ ¥ýãÔ¼÷› gpu.0.gpu‚28.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{?7S}£!‚(:ò ¥ýãÔÜÕ  disk.dm-1.out‚ 311.3203125 disk.sda.out‚ 311.3203125 disk.sda1.in‚0" proc.memory.rssMB‚ 2811.9140625 disk.drbd1716.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚45966.11572265625 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 31.1591796875& disk.nvme0n1.out‚45966.11572265625 disk.sda3.in‚ 0.1953125* proc.memory.percent‚17.162561416625977 disk.drbd1716.in‚0" disk.nvme0c0n1.in‚ 1167.7109375 network.sent‚ 2.0487036e+07* disk./.usagePercent‚16.641229331385325 disk.nvme0n1.in‚ 1167.7109375 disk.dm-0.in‚ 1167.7109375 disk.sda2.out‚0$ memory_percent‚91.30616188049316$ disk./.usageGB‚247.9663429260254 disk.dm-1.in‚ 0.1953125 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 311.3203125 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚45966.11572265625 network.recv‚ 2.416494e+06 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 1424.3984375 cpu‚25.362447449086773 proc.cpu.threads‚17'Zg%’¤!‚(:‡ ´ýãÔœ’Ç gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“T‚~¥!‚(:ó ´ýãÔÖÄú disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0$ memory_percent‚91.13202095031738( proc.memory.availableMB‚ 1452.9296875 cpu‚25.361757012396264* disk./.usagePercent‚16.642330419790095 disk.sda3.out‚ 311.83984375 disk.rbd1.in‚0% disk.nvme0n1.out‚46033.8037109375 network.sent‚ 2.0595217e+07 disk.drbd2265.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚17.162561416625977" disk.dm-0.out‚46033.8037109375 disk.sda3.in‚ 0.1953125" disk.nvme0c0n1.in‚ 1167.7109375' disk.nvme0c0n1.out‚46033.8037109375 disk.dm-0.in‚ 1167.7109375 disk.sda.out‚ 311.83984375 disk.drbd1716.out‚0 disk.rbd1.out‚ 31.2216796875 disk.nvme0n1.in‚ 1167.7109375 network.recv‚ 2.422133e+06 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2811.9140625 disk.sda2.out‚0 disk.drbd1716.in‚0% disk./.usageGB‚247.98274993896484 disk.dm-1.out‚ 311.83984375Éþ2¦!- _wandbruntime‚6683  _runtime‚6683´ÌÄŧ!¿ % lr-AdamW/pg1‚6.58288903540033e-05 % lr-AdamW/pg2‚6.58288903540033e-05 trainer/global_step‚8549 % _timestamp‚1.7884115805041833e+09 _runtime‚6683.491594832 _step‚701§dZʨ!Ä _runtime‚6683.491594832 _step‚701 % lr-AdamW/pg1‚6.58288903540033e-05 % lr-AdamW/pg2‚6.58288903540033e-05 trainer/global_step‚8549 % _timestamp‚1.7884115805041833e+09½àeN2©!- _wandbruntime‚6683  _runtime‚6683þsŒ³ª!áO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124609842 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498077437337088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498077437337088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7169461667840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚543837660813824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72710366 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7143716885441 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚498077419334144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 Wt„Ã4 JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚6683.492525678 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72710364 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505246881001984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚492154281535425 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485010564649984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485010547610624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7143062447685 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505247555367424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505247555367424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7520241736704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498077419334144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7169461667840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40994268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7520409554944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161583042 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40993821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚551358070368768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚8549 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161603080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884115805048912e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7170118030336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551357878610432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286192884 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124609844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚543837636873728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚492153610058309 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113704634 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚498077419334144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚6683.492525678 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72710364 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505246881001984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚492154281535425 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485010564649984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485010547610624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7143062447685 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505247555367424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505247555367424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7520241736704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498077419334144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7169461667840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40994268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7520409554944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161583042 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40993821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚551358070368768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚8549 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161603080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884115805048912e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7170118030336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551357878610432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286192884 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124609844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚543837636873728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚492153610058309 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113704634 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7521268656640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485065451741184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72718534 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72718532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124623858 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161621560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485065468780544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286245420 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505304727847424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚498133772372992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113716830 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543899311551488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286245420 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551420394754560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124623860 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40998298 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124623860 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7170281095680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40998748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7170937471488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286225354 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551420580208128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚6684.171183973 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113717282 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225354 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498133772372992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505304727847424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7521107143168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7144533320613 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505304053468672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161621560g_Plš®!ðM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7521107143168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7144533320613 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505304053468672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161621560 _step‚703 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7143878882649 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚492210002101157 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚8549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7170937471488 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124623858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543899287611392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7521268656640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485065451741184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72718534 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.curre’eV\nt‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72718532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124623858 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161621560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485065468780544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286245420 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505304727847424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚498133772372992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113716830 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543899311551488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286245420 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551420394754560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124623860 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40998298 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124623860 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7170281095680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40998748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7170937471488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286225354 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551420580208128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚6684.171183973 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113717282 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225354 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498133772372992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505304727847424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¿8¸\ï2¯!- _wandbruntime‚6684  _runtime‚6684XÒÚ]°!× 2loss/FBt4/assignment_loss‚0.004859304521232843 _step‚704 0loss/FRt3/assignment_loss‚1.6439691950690072 2loss/SRqqt4/assignment_loss‚1.6206344487145543 0loss/FRt4/assignment_loss‚2.6560412609577178 2loss/SRqqt3/assignment_loss‚0.9204382414591485 1loss/FBt2/assignment_loss‚0.34594788930068415 /loss/FRt4/detection_loss‚2.1874615007638933 epoch‚7 1loss/SRqqt2/assignment_loss‚1.007169865557681 2loss/FBt3/assignment_loss‚0.037291793107284706 /loss/FRt3/detection_loss‚1.7783624490477004 .loss/FRt1/detection_loss‚3.358381447584733 1loss/SRqqt4/detection_loss‚1.6835844341133321 /loss/FBt2/detection_loss‚1.8384746511777241 $ _timestamp‚1.788411581184828e+09 /loss/FRt2/assignment_loss‚2.982587352113904 .loss/FRt2/detection_loss‚2.203305463745942 1loss/SRqqt1/detection_loss‚2.2471189745094464 1loss/SRqqt3/detection_loss‚0.8484206630861106 %loss/total_loss‚45.33061613607842 _runtime‚6684.172153522 1loss/SRqqt2/detection_loss‚1.9696048323227011 trainer/global_step‚8549 0loss/FRt1/assignment_loss‚3.1975635756617007 0loss/FBt1/assignment_loss‚0.3411021794579158 /loss/FBt1/detection_loss‚2.6171405826296126 .loss/FBt3/detection_loss‚0.653598206046109 .loss/FBt4/detection_loss‚7.523696422576904 2loss/SRqqt1/assignment_loss‚1.6638614025533847N4#b±!Ü 0loss/FRt1/assignment_loss‚3.1975635756617007 0loss/FBt1/assignment_loss‚0.3411021794579158 /loss/FBt1/detection_loss‚2.6171405826296126 .loss/FBt3/detection_loss‚0.653598206046109 .loss/FBt4/detection_loss‚7.523696422576904 2loss/SRqqt1/assignment_loss‚1.6638614025533847 2loss/FBt4/assignment_loss‚0.004859304521232843 _step‚704 0loss/FRt3/assignment_loss‚1.6439691950690072 2loss/SRqqt4/assignment_loss‚1.6206344487145543 0loss/FRt4/assignment_loss‚2.6560412609577178 2loss/SRqqt3/assignment_loss‚0.9204382414591485 1loss/FBt2/assignment_loss‚0.34594788930068415 /loss/FRt4/detection_loss‚2.1874615007638933 epoch‚7 1loss/SRqqt2/assignment_loss‚1.007169865557681 2loss/FBt3/assignment_loss‚0.037291793107284706 /loss/FRt3/detection_loss‚1.7783624490477004 .loss/FRt1/detection_loss‚3.358381447584733 1loss/SRqqt4/detection_loss‚1.6835844341133321 /loss/FBt2/detection_loss‚1.8384746511777241 $ _timestamp‚1.788411581184828e+09 /loss/FRt2/assignment_loss‚2.982587352113904 .loss/FRt2/detection_loss‚2.203305463745942 1loss/SRqqt1/detection_loss‚2.2471189745094464 1loss/SRqqt3/detection_loss‚0.8484206630861106 %loss/total_loss‚45.33061613607842 _runtime‚6684.172153522 1loss/SRqqt2/detection_loss‚1.9696048323227011 trainer/global_step‚8549ÀÔ3¨Ä’²!‚(:‡ ÃýãÔ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cÑ(6e³!‚(:Ú ÃýãÔðâ²  disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 cpu‚25.3610944767119 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1953125 disk.nvme0n1.in‚ 1167.7109375 disk.sda.out‚ 312.32421875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.rbd1.in‚0 disk.dm-0.out‚46086.25) proc.memory.availableMB‚ 1479.48828125" proc.memory.rssMB‚ 2811.9140625 network.recv‚ 2.426281e+06 disk.sda3.out‚ 312.32421875 disk.drbd2266.out‚0 disk.nvme0n1.out‚46086.25% disk./.usageGB‚247.99849319458008 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.out‚46086.25 proc.cpu.threads‚17 disk.sda.in‚ 0.1953125 disk.sda1.out‚0 disk.dm-0.in‚ 1167.7109375 disk.dm-1.out‚ 312.32421875 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 31.2216796875 network.sent‚ 2.0601282e+07* disk./.usagePercent‚16.643386962883906 disk.sda2.in‚0 disk.drbd1716.out‚0$ memory_percent‚90.96992015838623* proc.memory.percent‚17.162561416625977´ö¹Œ´!‚(:… ÒýãÔû gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.72' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÚ ~µ!‚(:ó ÒýãÔÄßì disk.dm-0.in‚ 1167.7109375 proc.cpu.threads‚17( disk.nvme0c0n1.out‚46180.22509765625 disk.rbd1.in‚0 network.sent‚ 2.0709914e+07 network.recv‚ 2.432734e+06 disk.drbd2080.out‚0 disk.drbd2265.out‚0# disk.dm-0.out‚46180.22509765625 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2811.9140625 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 31.41015625 disk.nvme0n1.in‚ 1167.7109375& disk.nvme0n1.out‚46180.22509765625 disk.dm-1.out‚ 312.95703125( disk./.usagePercent‚16.6452949870366 disk.sda.out‚ 312.95703125 disk.drbd2080.in‚0$ memory_percent‚90.74714183807373) proc.memory.availableMB‚ 1515.98828125* proc.memory.percent‚17.162561416625977 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 312.95703125 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125% disk./.usageGB‚248.02692413330078 disk.rbd0.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 cpu‚25.360368152321353 disk.sda3.in‚ 0.1953125 disk.sda2.in‚0ÂU¬è’¶!‚(:‡ áýãÔƒÞ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ÚÝu·!‚(:ê áýãÔƒˆ¡  network.sent‚ 2.0716197e+07 network.recv‚ 2.436892e+06 disk.drbd1716.out‚0* proc.memory.percent‚17.162561416625977 disk.sda1.in‚0 disk.dm-0.in‚ 1167.7109375 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0$ memory_percent‚90.35491943359375 disk.nvme0n1.in‚ 1167.7109375& disk.nvme0n1.out‚46246.61181640625# disk.dm-0.out‚46246.61181640625 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 313.5546875 disk.sda.in‚ 0.1953125 disk.rbd1.out‚ 31.41015625 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.sda3.out‚ 313.5546875 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0# proc.memory.availableMB‚1580.25" proc.memory.rssMB‚ 2811.9140625* disk./.usagePercent‚16.646656179096155 disk.drbd2265.out‚0 disk.sda.out‚ 313.5546875 disk.sda2.out‚0$ disk./.usageGB‚248.0472068786621 disk.drbd2265.in‚0 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚46246.61181640625 disk.drbd1716.in‚0 cpu‚25.35967471232265 proc.cpu.threads‚17ˆd: 2¸!- _wandbruntime‚6720  _runtime‚6720¤\!ƹ!À & lr-AdamW/pg2‚6.414219382687585e-05 trainer/global_step‚8599 % _timestamp‚1.7884116170855505e+09 _runtime‚ 6720.07298613 _step‚705 & lr-AdamW/pg1‚6.414219382687585e-05T˺!Å % _timestamp‚1.7884116170855505e+09 _runtime‚ 6720.07298613 _step‚705 & lr-AdamW/pg1‚6.414219382687585e-05 & lr-AdamW/pg2‚6.414219382687585e-05 trainer/global_step‚8599Áü£C2»!- _wandbruntime‚6720  _runtime‚6720m[oWæ'¼!àO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚500895089279488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41220509 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487755771177984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508106193115648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7211103836160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125310644 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546920197697024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚487755754138624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7210447473664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73118764 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546920173756928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162509094 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494940323402833 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7183897787093 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚500895071276544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162529132 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508105518750208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554483465187328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7184552224849 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚706 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494939651925717 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7210447473664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500895071276544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125310642 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _runtime‚6720.073899683 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125310642 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚508106193115648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7211103836160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287839776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73118766 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287819736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125310644 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7563267490304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500895089279488 $ _timestamp‚1.788411617086228e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114339735 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114339273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41220969 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554483279720448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508105518750208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7563105963520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚500895089279488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41220509 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487755771177984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508106193115648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7211103836160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125310644 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546920197697024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚487755754138624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7210447473664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73118764 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546920173756928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162509094 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494940323402833 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7183897787093 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚500895071276544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162529132 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508105518750208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554483465187328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7184552224849 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚706 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494939651925717 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7210447473664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500895071276544 QCDeviceStatsMo.;ê¬nitor.on_train_batch_start/allocation.large_pool.freed‚ 125310642 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _runtime‚6720.073899683 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125310642 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚508106193115648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7211103836160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287839776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73118766 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287819736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125310644 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7563267490304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500895089279488 $ _timestamp‚1.788411617086228e+09Â`Šœ2¾!- _wandbruntime‚6720  _runtime‚6720,ä[bñ&¿!ëM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚500951424315392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508162691216896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚500951442318336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41225493 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6720.749188471 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162547612 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162527548 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114351970 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852206 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚500951424315392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚554545788664832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7564125683712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487810658269184 _step‚707 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125324660 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546981848434688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚546981824494592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494996043968565 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7185368660021 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73126934 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162547612 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚8599 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884116177610493e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125324658 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125324658 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494995372491241 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114352427 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554545974118400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41225038 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508163365595648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚500951442318336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7563964170240 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7211266901504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287852206 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125324660 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7184714222057 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414EøDö&À!ðM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494996043968565 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7185368660021 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73126934 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162547612 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚8599 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884116177610493e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125324658 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125324658 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494995372491241 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114352427 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554545974118400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41225038 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508163365595648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚500951442318336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7563964170240 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7211266901504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287852206 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125324660 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7184714222057 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚500951424315392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508162691216896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚500951442318336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41225493 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6720.749188471 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162547612 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162527548 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114351970 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852206 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚500951424315392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚554545788664832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7564125683712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487810658269184 _step‚707 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125324660 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546981848434688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚546981824494592Ãt?XZ2Á!- _wandbruntime‚6720  _runtime‚6720cräYÂ!Ó 1loss/FBt2/assignment_loss‚0.07980773001904709 0loss/SRqqt4/detection_loss‚1.650071302221881 trainer/global_step‚8599 2loss/SRqqt2/assignment_loss‚1.6626189224421979 1loss/SRqqt4/assignment_loss‚1.249484538756035 1loss/FBt3/assignment_loss‚0.08642791568805572 .loss/FRt1/detection_loss‚2.874855191418619 /loss/FBt4/detection_loss‚3.1741277277469635 % _timestamp‚1.7884116177629325e+09 1loss/FBt4/assignment_loss‚0.42741330941983807 _step‚708 2loss/SRqqt1/assignment_loss‚1.5877568889409304 .loss/FRt4/detection_loss‚2.405806446528953 1loss/SRqqt1/detection_loss‚2.3808376314118505 1loss/SRqqt3/detection_loss‚0.9903566538268933 .loss/FBt1/detection_loss‚2.472161151468754 /loss/FRt2/assignment_loss‚3.240756246058837 0loss/FRt3/assignment_loss‚1.7279662433124723 _runtime‚6720.750331537 epoch‚7 /loss/FRt1/assignment_loss‚3.090556067047697 2loss/SRqqt3/assignment_loss‚1.2871186272823252 .loss/FRt2/detection_loss‚2.340353454584695 /loss/FRt3/detection_loss‚1.9278475587211905 .loss/FBt2/detection_loss‚2.550619515776634 /loss/FBt3/detection_loss‚0.5378716672253278 %loss/total_loss‚42.73332763781106 0loss/FRt4/assignment_loss‚2.7676418112671892 0loss/FBt1/assignment_loss‚0.1942419964034343 1loss/SRqqt2/detection_loss‚2.0266290402412412&a‹¤^Ã!Ø 2loss/SRqqt1/assignment_loss‚1.5877568889409304 .loss/FRt4/detection_loss‚2.405806446528953 1loss/SRqqt1/detection_loss‚2.3808376314118505 1loss/SRqqt3/detection_loss‚0.9903566538268933 .loss/FBt1/detection_loss‚2.472161151468754 /loss/FRt2/assignment_loss‚3.240756246058837 0loss/FRt3/assignment_loss‚1.7279662433124723 _runtime‚6720.750331537 epoch‚7 /loss/FRt1/assignment_loss‚3.090556067047697 2loss/SRqqt3/assignment_loss‚1.2871186272823252 .loss/FRt2/detection_loss‚2.340353454584695 /loss/FRt3/detection_loss‚1.9278475587211905 .loss/FBt2/detection_loss‚2.550619515776634 /loss/FBt3/detection_loss‚0.5378716672253278 %loss/total_loss‚42.73332763781106 0loss/FRt4/assignment_loss‚2.7676418112671892 0loss/FBt1/assignment_loss‚0.1942419964034343 1loss/SRqqt2/detection_loss‚2.0266290402412412 1loss/FBt2/assignment_loss‚0.07980773001904709 0loss/SRqqt4/detection_loss‚1.650071302221881 trainer/global_step‚8599 2loss/SRqqt2/assignment_loss‚1.6626189224421979 1loss/SRqqt4/assignment_loss‚1.249484538756035 1loss/FBt3/assignment_loss‚0.08642791568805572 .loss/FRt1/detection_loss‚2.874855191418619 /loss/FBt4/detection_loss‚3.1741277277469635 % _timestamp‚1.7884116177629325e+09 1loss/FBt4/assignment_loss‚0.42741330941983807 _step‚708Ä¢šg8’Ä!‚(:‡ ðýãÔœ” gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.951' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SN:{Å!‚(:ð ðýãÔ¹é— # disk.dm-0.out‚46296.14794921875 disk.drbd2265.out‚0$ memory_percent‚89.82720375061035( proc.memory.availableMB‚ 1666.7109375 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 1167.7109375 cpu‚25.35894342870641 network.sent‚ 2.0720563e+07" disk.nvme0c0n1.in‚ 1167.7109375 disk.rbd1.out‚ 31.47265625 disk.sda.out‚ 314.1640625% disk./.usageGB‚248.06790161132812 disk.dm-1.in‚ 0.1953125* proc.memory.percent‚17.162561416625977 proc.cpu.threads‚17 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 314.1640625 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚46296.14794921875 disk.dm-1.out‚ 314.1640625 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1167.7109375 disk.sda.in‚ 0.1953125" proc.memory.rssMB‚ 2811.9140625* disk./.usagePercent‚16.648045019969405 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚46296.14794921875 disk.drbd1716.out‚0 network.recv‚ 2.439906e+06 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0¦3’’Æ!‚(:‡ ÿýãÔñ¬õ gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:­X|Ç!‚(:ñ ÿýãÔ£…þ disk.sda3.out‚ 314.5234375 disk.drbd2080.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚46350.71435546875 disk.drbd1716.out‚0 disk.dm-1.out‚ 314.5234375 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.sda.out‚ 314.5234375 disk.drbd2266.out‚0 disk.sda1.out‚0* disk./.usagePercent‚16.648922357789246% disk./.usageGB‚248.08097457885742 disk.dm-0.in‚ 1167.7109375 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1167.7109375 network.sent‚ 2.0829251e+07# disk.dm-0.out‚46350.71435546875 disk.dm-1.in‚ 0.1953125& disk.nvme0n1.out‚46350.71435546875* proc.memory.percent‚17.162561416625977 disk.drbd2265.out‚0( proc.memory.availableMB‚ 1666.7109375 cpu‚25.358231810268993 network.recv‚ 2.446244e+06 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.rbd1.in‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 31.47265625$ memory_percent‚89.82720375061035" proc.memory.rssMB‚ 2811.9140625 proc.cpu.threads‚17U?2È!- _wandbruntime‚6756  _runtime‚67565uôZÆÉ!À trainer/global_step‚8649 % _timestamp‚1.7884116531496058e+09 _runtime‚ 6756.13704153 _step‚709 & lr-AdamW/pg1‚6.247079110955025e-05 & lr-AdamW/pg2‚6.247079110955025e-05eÏ/pËÊ!Å _step‚709 & lr-AdamW/pg1‚6.247079110955025e-05 & lr-AdamW/pg2‚6.247079110955025e-05 trainer/global_step‚8649 % _timestamp‚1.7884116531496058e+09 _runtime‚ 6756.13704153År"˜u2Ë!- _wandbruntime‚6756  _runtime‚6756L+ïÌ!áO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510964156498432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114975237 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510964830863872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510964156498432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114974803 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510964830863872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126011442 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7251433279488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550002710640128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚490500977705984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163435146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73527164 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73527166 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7252089641984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _step‚710 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41448071 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚490500960666624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163455184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557609074167808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503712723218944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚›:2Åø0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289466628 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41447639 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7225387564257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163435146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289446588 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7606188546560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126011442 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126011444 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503712723218944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503712741221888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7252089641984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚6756.138009015 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884116531502552e+09çt>)ì'Í!æO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚550002734580224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41447639 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7225387564257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163435146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289446588 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7606188546560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126011442 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126011444 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503712723218944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503712741221888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7252089641984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚6756.138009015 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884116531502552e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510964156498432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114975237 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510964830863872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510964156498432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114974803 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510964830863872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126011442 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7251433279488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550002710640128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚490500977705984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163435146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73527164 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73527166 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7252089641984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _step‚710 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41448071 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚490500960666624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163455184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557609074167808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503712723218944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289466628 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499124 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41452592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73535334 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126025458 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚490555864797184 trainer/global_step‚8649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503769094260736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114987926 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚497782085835973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503769094260736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7252909083136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126025460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚557671410177024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41452136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚490555881836544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289479058 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163453600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚511022003343872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚511021328965120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163473664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7225549561465 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126025458 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7252909083136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚711 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114987468 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126025460 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289479058 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7226203999429 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7252252707328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511022003343872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163453600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚6756.813090309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163473664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503769076257792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550064385317888 $ _timestamp‚1.788411653825123e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73535332 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚503769076257792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289499124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557671597727744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚550064361377792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 "}×Ð!ïM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503769094260736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7252909083136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126025460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚557671410177024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41452136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚490555881836544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289479058 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163453600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚511022003343872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚511021328965120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163473664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7225549561465 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126025458 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7252909083136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚711 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114987468 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126025460 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289479058 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7226203999429 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7252252707328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511022003343872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163453600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚6756.813090309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163473664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503769076257792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550064385317888 $ _timestamp‚1.788411653825123e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73535332 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚503769076257792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289499124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557671597727744 _KDeviceStatsMonitor.on_trØ”J ain_batch_end/inactive_split_bytes.large_pool.freed‚550064361377792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7607048799232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7607212409856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499124 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41452592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73535334 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126025458 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚490555864797184 trainer/global_step‚8649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503769094260736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114987926 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚497782085835973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844Çvˉ2Ñ!- _wandbruntime‚6756  _runtime‚6756ÜûHXZÒ!Ô /loss/FRt1/detection_loss‚3.2451033615148983 /loss/SRqqt3/detection_loss‚1.08294891501809 2loss/SRqqt2/assignment_loss‚1.2897296518680046 1loss/SRqqt4/detection_loss‚1.9067582562565804 /loss/FBt4/detection_loss‚2.2422564387321473 trainer/global_step‚8649 /loss/FRt4/assignment_loss‚2.284176482443224 1loss/FBt4/assignment_loss‚0.12872125072753987 0loss/SRqqt1/detection_loss‚1.960320251143497 .loss/FBt1/detection_loss‚3.323389971256256 %loss/total_loss‚42.17156399380177 2loss/SRqqt1/assignment_loss‚1.8656063479087923 2loss/SRqqt3/assignment_loss‚1.0074601855193233 0loss/FBt3/detection_loss‚0.37138746847296883 $ _timestamp‚1.788411653826935e+09 0loss/FRt3/assignment_loss‚1.7689131059144672 .loss/FRt4/detection_loss‚2.304602567041129 1loss/SRqqt2/detection_loss‚1.9794973622668872 .loss/FBt2/detection_loss‚2.412330141434303 _runtime‚6756.814307245 _step‚712 2loss/FBt2/assignment_loss‚0.038145968780064266 /loss/FRt3/detection_loss‚2.0074082169387686 2loss/SRqqt4/assignment_loss‚1.2914336340215344 0loss/FBt1/assignment_loss‚0.7511115368688479 epoch‚7 /loss/FRt2/detection_loss‚2.2079448768641905 /loss/FRt1/assignment_loss‚3.112856376629609 /loss/FRt2/assignment_loss‚3.381544658115932 1loss/FBt3/assignment_loss‚0.20791696806471052«yŸ-_Ó!Ù 0loss/FBt1/assignment_loss‚0.7511115368688479 epoch‚7 /loss/FRt2/detection_loss‚2.2079448768641905 /loss/FRt1/assignment_loss‚3.112856376629609 /loss/FRt2/assignment_loss‚3.381544658115932 1loss/FBt3/assignment_loss‚0.20791696806471052 /loss/FRt1/detection_loss‚3.2451033615148983 /loss/SRqqt3/detection_loss‚1.08294891501809 2loss/SRqqt2/assignment_loss‚1.2897296518680046 1loss/SRqqt4/detection_loss‚1.9067582562565804 /loss/FBt4/detection_loss‚2.2422564387321473 trainer/global_step‚8649 /loss/FRt4/assignment_loss‚2.284176482443224 1loss/FBt4/assignment_loss‚0.12872125072753987 0loss/SRqqt1/detection_loss‚1.960320251143497 .loss/FBt1/detection_loss‚3.323389971256256 %loss/total_loss‚42.17156399380177 2loss/SRqqt1/assignment_loss‚1.8656063479087923 2loss/SRqqt3/assignment_loss‚1.0074601855193233 0loss/FBt3/detection_loss‚0.37138746847296883 $ _timestamp‚1.788411653826935e+09 0loss/FRt3/assignment_loss‚1.7689131059144672 .loss/FRt4/detection_loss‚2.304602567041129 1loss/SRqqt2/detection_loss‚1.9794973622668872 .loss/FBt2/detection_loss‚2.412330141434303 _runtime‚6756.814307245 _step‚712 2loss/FBt2/assignment_loss‚0.038145968780064266 /loss/FRt3/detection_loss‚2.0074082169387686 2loss/SRqqt4/assignment_loss‚1.2914336340215344Èÿ·Þº’Ô!‚(:‡ ŽþãÔ™Ëë gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.699' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õö£fÕ!‚(:Û ŽþãÔêÍœ  cpu‚25.357541305407103" disk.nvme0c0n1.in‚ 1167.7109375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1167.7109375 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 network.recv‚ 2.450713e+06* disk./.usagePercent‚16.650301726383734$ disk./.usageGB‚248.1015281677246 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2811.9140625 disk.drbd2265.out‚0 disk.sda.out‚315.25* proc.memory.percent‚17.162561416625977 network.sent‚ 2.0835081e+07 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚46428.23046875# disk.nvme0n1.out‚46428.23046875 disk.dm-1.out‚315.25 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1167.7109375 disk.drbd2266.in‚0$ memory_percent‚89.82760906219482 disk.rbd1.out‚ 31.5986328125 disk.dm-0.out‚46428.23046875 disk.sda3.in‚ 0.1953125 disk.sda3.out‚315.25) proc.memory.availableMB‚ 1666.64453125ýHÝ+Ö!‚(:’ þãÔ¬Þí gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.291' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.458199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jñz×!‚(:ï þãÔšéå  disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 1167.7109375) proc.memory.availableMB‚ 1727.40234375$ disk./.usageGB‚248.1371078491211 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 1167.7109375& disk.nvme0n1.out‚46493.04638671875$ memory_percent‚89.45677280426025 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚46493.04638671875 cpu‚25.35686675122042 proc.cpu.threads‚17 disk.rbd1.out‚ 31.5986328125 disk.nvme0n1.in‚ 1167.7109375 disk.sda.out‚ 315.671875 disk.rbd0.in‚0 disk.sda1.out‚0# disk.dm-0.out‚46493.04638671875" proc.memory.rssMB‚ 2811.9140625 network.sent‚ 2.0944281e+07 network.recv‚ 2.457519e+06* disk./.usagePercent‚16.652689508655595 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 315.671875 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.162561416625977 disk.sda3.out‚ 315.671875úä€u2Ø!- _wandbruntime‚6792  _runtime‚6792q)ÃÅÈÙ! trainer/global_step‚8699 $ _timestamp‚1.788411689411008e+09 _runtime‚6792.398436044 _step‚713 ' lr-AdamW/pg1‚6.0815030519847716e-05 ' lr-AdamW/pg2‚6.0815030519847716e-05”Ÿd„ÍÚ!Ç ' lr-AdamW/pg2‚6.0815030519847716e-05 trainer/global_step‚8699 $ _timestamp‚1.788411689411008e+09 _runtime‚6792.398436044 _step‚713 ' lr-AdamW/pg1‚6.0815030519847716e-05ÉŽ“?2Û!- _wandbruntime‚6792  _runtime‚6792”2ã¶ç'Ü!áO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚513823468612096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚506530393164288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553085271463424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚500511735660533 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41676070 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513822794246656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7293075447808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164381236 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126712242 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560734632253952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚506530393164288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164381236 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚500512407137649 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493246167194624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚506530375161344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291093480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7649384730624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126712244 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513822794246656 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115611180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115611636 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164361198 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7649537868800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291073440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7292419085312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7265568465909 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚560734809332224 JòhÛì'Ý!æO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126712244 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513822794246656 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115611180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115611636 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164361198 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7649537868800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291073440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7292419085312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7265568465909 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚560734809332224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚506530393164288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553085271463424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚500511735660533 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41676070 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513822794246656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7293075447808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164381236 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126712242 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560734632253952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚506530393164288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164381236 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚500512407137649 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493246167194624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚506530375161344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291093480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7649384730624ÊÂÒò2Þ!- _wandbruntime‚6793  _runtime‚6793¯âšÚz ß!ëM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560797330119168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164379652 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164379652 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553146922201088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500568127703381 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291125976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73943732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 RBDeviceStatsMonitor.on_train_b¨ïÚwatch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7293894888960 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115623915 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500567456226057 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚493301088364544 % _timestamp‚1.7884116900911698e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553146898260992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126726260 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164399716 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41680183 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚493301071325184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7266384900873 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126726258 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41680652 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7293238513152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚715 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506586728200192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7293894888960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73943734 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7650407918080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513879966713344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506586728200192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚506586746203136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126726258 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126726260ë&š ö&à!ðM trainer/global_step‚8699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291125976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73943732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7293894888960 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115623915 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500567456226057 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚493301088364544 % _timestamp‚1.7884116900911698e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553146898260992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126726260 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164399716 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41680183 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚493301071325184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7266384900873 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126726258 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41680652 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7293238513152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚715 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506586728200192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7293894888960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73943734 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7650407918080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513879966713344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506586728200192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚506586746203136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126726258 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126726260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560797330119168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164379652 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164379652 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553146922201088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500568127703381 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ËáÏî;2á!- _wandbruntime‚6793  _runtime‚67938u›ŽTâ!Î 1loss/SRqqt4/assignment_loss‚1.249918307315696 2loss/SRqqt2/assignment_loss‚1.1661946386942323 /loss/FRt2/detection_loss‚1.9459955808649883 .loss/FBt2/detection_loss‚2.554538243346744 0loss/FRt3/assignment_loss‚1.4750458916028342 1loss/SRqqt1/assignment_loss‚2.089356999993324 1loss/FBt1/assignment_loss‚0.23270755974036464 0loss/FBt4/assignment_loss‚0.2037722464052162 0loss/SRqqt1/detection_loss‚2.570236965417862 0loss/SRqqt4/detection_loss‚1.607087135038994 trainer/global_step‚8699 % _timestamp‚1.7884116900927777e+09 0loss/FRt2/assignment_loss‚2.8302271135828714 2loss/SRqqt3/assignment_loss‚0.8682323189388061 1loss/FBt3/assignment_loss‚0.12335268418727778 /loss/FRt3/detection_loss‚1.6752022177808814 /loss/FBt1/detection_loss‚2.0650033300573174 /loss/FRt1/assignment_loss‚2.658518005300451 /loss/FRt1/detection_loss‚2.9759518835279675 1loss/SRqqt3/detection_loss‚1.1138176429888298 .loss/FBt3/detection_loss‚0.500106843537651 .loss/FBt4/detection_loss‚2.886481557573591 _runtime‚ 6793.08016933 0loss/FBt2/assignment_loss‚0.2966217517645823 /loss/FRt4/detection_loss‚2.4903209858470494 $loss/total_loss‚40.0743817506449 _step‚716 .loss/FRt4/assignment_loss‚2.83884531127082 0loss/SRqqt2/detection_loss‚1.656846535866553 epoch‚7æíøYã!Ó 0loss/FRt3/assignment_loss‚1.4750458916028342 1loss/SRqqt1/assignment_loss‚2.089356999993324 1loss/FBt1/assignment_loss‚0.23270755974036464 0loss/FBt4/assignment_loss‚0.2037722464052162 0loss/SRqqt1/detection_loss‚2.570236965417862 0loss/SRqqt4/detection_loss‚1.607087135038994 trainer/global_step‚8699 % _timestamp‚1.7884116900927777e+09 0loss/FRt2/assignment_loss‚2.8302271135828714 2loss/SRqqt3/assignment_loss‚0.8682323189388061 1loss/FBt3/assignment_loss‚0.12335268418727778 /loss/FRt3/detection_loss‚1.6752022177808814 /loss/FBt1/detection_loss‚2.0650033300573174 /loss/FRt1/assignment_loss‚2.658518005300451 /loss/FRt1/detection_loss‚2.9759518835279675 1loss/SRqqt3/detection_loss‚1.1138176429888298 .loss/FBt3/detection_loss‚0.500106843537651 .loss/FBt4/detection_loss‚2.886481557573591 _runtime‚ 6793.08016933 0loss/FBt2/assignment_loss‚0.2966217517645823 /loss/FRt4/detection_loss‚2.4903209858470494 $loss/total_loss‚40.0743817506449 _step‚716 .loss/FRt4/assignment_loss‚2.83884531127082 0loss/SRqqt2/detection_loss‚1.656846535866553 epoch‚7 1loss/SRqqt4/assignment_loss‚1.249918307315696 2loss/SRqqt2/assignment_loss‚1.1661946386942323 /loss/FRt2/detection_loss‚1.9459955808649883 .loss/FBt2/detection_loss‚2.554538243346744ÌW¦Ñ’ä!‚(:‡ ¬þãÔ¸Û™ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.083' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥»¥}å!‚(:ò ¬þãÔŽ¥€ & disk.nvme0n1.out‚46558.05029296875 disk.dm-0.in‚ 1167.7109375 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2811.9140625 disk.sda.out‚ 316.16015625 disk.sda2.in‚0 disk.drbd2080.in‚0 network.recv‚ 2.461322e+06 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 316.16015625 disk.sda1.out‚0 disk.drbd1716.in‚0# disk.dm-0.out‚46558.05029296875 disk.dm-1.in‚ 0.1953125 cpu‚25.35615356726059* disk./.usagePercent‚16.651956559085065 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd2265.in‚0& proc.memory.availableMB‚ 1814.90625 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 31.6611328125 disk.nvme0n1.in‚ 1167.7109375$ memory_percent‚88.92269134521484 network.sent‚ 2.0949589e+07$ disk./.usageGB‚248.1261863708496 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 316.16015625 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚46558.05029296875* proc.memory.percent‚17.162561416625977 disk.sda3.in‚ 0.1953125›i’Úæ!‚(:’ »þãÔÿÈâ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚183.593' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.718599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©ßÌÿdç!‚(:Ù »þãÔ…âó  network.sent‚ 2.1058241e+07 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 1167.7109375 disk.drbd1716.in‚0' proc.memory.availableMB‚ 1814.671875 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 46624.1171875 disk.rbd1.in‚0 disk.rbd1.out‚ 31.6611328125% disk./.usageGB‚248.16012954711914 disk.dm-0.out‚ 46624.1171875 disk.drbd2265.in‚0 disk.sda.out‚316.625" proc.memory.rssMB‚ 2811.9140625* disk./.usagePercent‚16.654234514124735 disk.dm-0.in‚ 1167.7109375 disk.rbd0.in‚0 disk.drbd1716.out‚0" disk.nvme0n1.out‚ 46624.1171875 disk.dm-1.out‚316.625 disk.rbd0.out‚0* proc.memory.percent‚17.162561416625977 cpu‚25.35548250245571 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1167.7109375 disk.drbd2265.out‚0 disk.sda3.out‚316.625$ memory_percent‚88.92412185668945 network.recv‚ 2.467414e+06 disk.dm-1.in‚ 0.1953125½Rè<è!‚(:… ÊþãÔÙË‹ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓqYûmé!‚(:â ÊþãÔóê  disk.dm-1.out‚ 317.4140625* proc.memory.percent‚17.162561416625977 network.sent‚ 2.1064076e+07 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 disk.dm-0.out‚ 46712.421875 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2811.9140625 disk.rbd0.in‚0! disk.nvme0n1.out‚ 46712.421875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0' proc.memory.availableMB‚ 1879.671875 cpu‚25.354782931789384 disk.drbd2266.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1167.7109375$ memory_percent‚88.52739334106445 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 317.4140625 disk.sda3.out‚ 317.4140625 disk.rbd1.out‚ 31.787109375 disk.dm-0.in‚ 1167.7109375 disk.sda.in‚ 0.1953125 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1953125 network.recv‚ 2.471636e+06% disk./.usageGB‚248.18685913085938 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1167.7109375* disk./.usagePercent‚16.656028358917155 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 46712.421875Xšî2ê!- _wandbruntime‚6828  _runtime‚6828.¤äÇë!Á _runtime‚6828.856997717 _step‚717 & lr-AdamW/pg1‚5.917525711579332e-05 & lr-AdamW/pg2‚5.917525711579332e-05 trainer/global_step‚8749 % _timestamp‚1.7884117258695457e+09±ö))Ìì!Æ % _timestamp‚1.7884117258695457e+09 _runtime‚6828.856997717 _step‚717 & lr-AdamW/pg1‚5.917525711579332e-05 & lr-AdamW/pg2‚5.917525711579332e-05 trainer/global_step‚8749Í£[„2í!- _wandbruntime‚6828  _runtime‚6828lÊQ‘@î!àO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292700292 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚509348045106688 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116248227 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74343964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚509348027103744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚516682106360320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚509348027103744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚516682106360320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556167808346624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127413042 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41904698 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7692620732928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7306403805317 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127413042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74343966 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚556167784406528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7307058243073 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚495991390761984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚516681431994880 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516681431994880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116248664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 % _timestamp‚1.7884117258702543e+09 _runtime‚ 6828.85794328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚509348045106688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚563860429079552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚516682106360320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556167808346624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127413042 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41904698 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7692620732928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7306403805317 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127413042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74343966 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚556167784406528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7307058243073 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚495991390761984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚516681431994880 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516681431994880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116248664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 % _timestamp‚1.7884117258702543e+09 _runtime‚ 6828.85794328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚509348045106688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚563860429079552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7693317803008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127427058 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116260886 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74352134 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚509404380142592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127427060 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚6829.773563712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7307874678245 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚516739278840320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚509404398145536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752828 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165305704 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496046294892544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7334224318976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292732762 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41909212 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165325768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292732762 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7334880694784XmÁö&ò!ðM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚509404380142592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚719 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41909212 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165325768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292732762 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7334880694784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7334224318976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚8749 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127427058 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚563922752947200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚556229435144192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503353498093465 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127427060 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165325768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚516739278840320 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7693317803008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127427058 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116260886 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74352134 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚509404380142592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127427060 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚6829.773563712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7307874678245 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚516739278840320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚509404398145536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752828 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165305704 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496046294892544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7334224318976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292732762 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7347239144725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7375047059456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566986206354432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559250345229824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74752366 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166233340 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74752364 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7735861124608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚506083819395349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166213302 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566986016693248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7735695403520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128113842 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519540069743104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42131676 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7735861124608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚506083819395349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166213302 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566986016693248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7735695403520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128113842 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519540069743104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42131676 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7347239144725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7375047059456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566986206354432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559250345229824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74752366 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166233340 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928Òø­P2€"- _wandbruntime‚6866  _runtime‚6866â *°"éM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512222032084992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7375866500608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512222032084992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116896756 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519597916588544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166251820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7348710017653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42136222 _runtime‚6866.247057755 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚506140211438197 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498791501420544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498791484381184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166231756 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166251820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512222050087936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74760532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚559311972027Èì©Ü?392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 # _timestamp‚1.78841176325917e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519597242209792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128127860 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166231756 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7375210124800 trainer/global_step‚8799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294379680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7736567835648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7375866500608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506139539960873 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚723 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116896313 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7348055579689 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512222050087936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42135781 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567048539863040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294359614 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294359614 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74760534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128127858 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320¤4Z¡ô&‚"îM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚567048723219456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166251820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512222050087936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74760532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚559311972027392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 # _timestamp‚1.78841176325917e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519597242209792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128127860 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166231756 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7375210124800 trainer/global_step‚8799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294379680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7736567835648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7375866500608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506139539960873 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚723 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116896313 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7348055579689 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512222050087936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42135781 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567048539863040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294359614 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294359614 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74760534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128127858 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512222032084992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7375866500608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512222032084992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116896756 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519597916588544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166251820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7348710017653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42136222 _runtime‚6866.247057755 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚506140211438197 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498791501420544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498791484381184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166231756 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚508870532739873 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚508869861262757 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7388728921889 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7416032865280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚570111644313600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128814642 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚570111840266240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522398707491328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295953996 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42358174 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514983348991488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514983330988544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522398707491328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514983348991488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117519405 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚522399381856768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7388074484133 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7778958153216 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167139354 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7416032865280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974036 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚726!šÍs‘"äO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7415376502784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75160764 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_kCjw&train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167159392 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117518938 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501481786778624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128814644 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128814644 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42358639 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚508870532739873 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚508869861262757 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7388728921889 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7416032865280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚570111644313600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128814642 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚570111840266240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522398707491328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295953996 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42358174 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514983348991488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514983330988544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522398707491328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514983348991488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117519405 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚522399381856768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7388074484133 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7778958153216 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167139354 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7416032865280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974036 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚726 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128814642 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÖûÿÞò2’"- _wandbruntime‚6902  _runtime‚6902lÒO‘ñ&“"ëM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570174350342144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75168932 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167177872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522455879958016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295986466 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562394508910592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚515039684027392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128828658 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42362640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚727 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚508926253305605 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508925581828281 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515039702030336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167157808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7416195930624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562394532850688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7416852306432 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117531572 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7779660172288 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7388890919097 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7416195930624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7389545357061 trainer/global_step‚8849 % _timestamp‚1.7884117998520997e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167177872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522456554336768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167157808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75168934 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522456554336768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7416852306432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7779817491456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128828660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42363096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296006532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚570174169082880 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096F¬G´ö&”"ðM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42363096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296006532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚570174169082880 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570174350342144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75168932 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167177872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522455879958016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295986466 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562394508910592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚515039684027392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128828658 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42362640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚727 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚508926253305605 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508925581828281 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515039702030336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167157808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7416195930624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562394532850688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7416852306432 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117531572 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7779660172288 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7388890919097 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7416195930624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7389545357061 trainer/global_step‚8849 % _timestamp‚1.7884117998520997e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167177872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522456554336768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167157808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75168934 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522456554336768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7416852306432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7779817491456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128828660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0×>´è2•"- _wandbruntime‚6902  _runtime‚6902B<¶”[–"Õ /loss/FRt2/detection_loss‚2.2318636968415126 /loss/FRt3/detection_loss‚2.0797925315252166 1loss/SRqqt4/detection_loss‚1.6324829272925854 -loss/FBt4/detection_loss‚2.37567575275898 1loss/FBt3/assignment_loss‚0.13973015475477588 /loss/FRt2/assignment_loss‚2.883599378961198 0loss/FRt3/assignment_loss‚1.6963674048098123 2loss/SRqqt4/assignment_loss‚1.6804660855993336 /loss/FRt1/assignment_loss‚2.520339752237002 /loss/FRt4/assignment_loss‚2.143554989496867 2loss/SRqqt3/assignment_loss‚1.0403925517928743 1loss/FBt2/assignment_loss‚0.11012611298688821 1loss/FBt4/assignment_loss‚0.23251518925341466 .loss/FRt4/detection_loss‚2.474556642770767 0loss/SRqqt1/detection_loss‚2.392085237428546 1loss/SRqqt3/detection_loss‚1.0045674632466028 2loss/SRqqt2/assignment_loss‚1.4746633601650871 /loss/FRt1/detection_loss‚3.4522428438067436 $ _timestamp‚1.788411799854205e+09 _step‚728 /loss/FBt3/detection_loss‚0.5934196871052357 _runtime‚6902.841567951 /loss/FBt2/detection_loss‚2.7929466366767883 epoch‚7 trainer/global_step‚8849 2loss/SRqqt1/assignment_loss‚1.4655359104217496 2loss/FBt1/assignment_loss‚0.012631529879096584 0loss/SRqqt2/detection_loss‚1.779514054733294 .loss/FBt1/detection_loss‚2.251404106616974 &loss/total_loss‚40.460474001161344.Êeƒ`—"Ú /loss/FRt3/detection_loss‚2.0797925315252166 1loss/SRqqt4/detection_loss‚1.6324829272925854 -loss/FBt4/detection_loss‚2.37567575275898 1loss/FBt3/assignment_loss‚0.13973015475477588 /loss/FRt2/assignment_loss‚2.883599378961198 0loss/FRt3/assignment_loss‚1.6963674048098123 2loss/SRqqt4/assignment_loss‚1.6804660855993336 /loss/FRt1/assignment_loss‚2.520339752237002 /loss/FRt4/assignment_loss‚2.143554989496867 2loss/SRqqt3/assignment_loss‚1.0403925517928743 1loss/FBt2/assignment_loss‚0.11012611298688821 1loss/FBt4/assignment_loss‚0.23251518925341466 .loss/FRt4/detection_loss‚2.474556642770767 0loss/SRqqt1/detection_loss‚2.392085237428546 1loss/SRqqt3/detection_loss‚1.0045674632466028 2loss/SRqqt2/assignment_loss‚1.4746633601650871 /loss/FRt1/detection_loss‚3.4522428438067436 $ _timestamp‚1.788411799854205e+09 _step‚728 /loss/FBt3/detection_loss‚0.5934196871052357 _runtime‚6902.841567951 /loss/FBt2/detection_loss‚2.7929466366767883 epoch‚7 trainer/global_step‚8849 2loss/SRqqt1/assignment_loss‚1.4655359104217496 2loss/FBt1/assignment_loss‚0.012631529879096584 0loss/SRqqt2/detection_loss‚1.779514054733294 .loss/FBt1/detection_loss‚2.251404106616974 &loss/total_loss‚40.460474001161344 /loss/FRt2/detection_loss‚2.2318636968415126ØâT‚ÚK˜"‚(:‡ ¤ÿãÔ·»  gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.mg̲jGemoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.951' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÉ#Ê™"‚(:ö ¤ÿãÔÜج  disk.drbd2265.out‚0 disk.sda.out‚ 320.79296875 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 cpu‚25.350792668214027 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.sda1.out‚0! disk.nvme0n1.in‚ 1187.62109375$ memory_percent‚90.18404483795166 disk.sda3.in‚ 0.1953125* proc.memory.percent‚17.162561416625977" disk.dm-0.out‚47268.2646484375" proc.memory.rssMB‚ 2811.9140625 disk.dm-1.out‚ 320.79296875% disk.nvme0n1.out‚47268.2646484375 proc.cpu.threads‚17 network.sent‚ 2.1304787e+07* disk./.usagePercent‚16.664857546758036% disk./.usageGB‚248.31842041015625 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚47268.2646484375 disk.dm-1.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚ 32.1240234375) proc.memory.availableMB‚ 1608.24609375 network.recv‚ 2.49993e+06 disk.sda3.out‚ 320.79296875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 1187.62109375êmx}š"‚(:’ ³ÿãÔÞŠÆ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.051' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.410199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C|üw›"‚(:ì ³ÿãÔÁùÌ  disk.rbd1.in‚0" proc.memory.rssMB‚ 2811.9140625 network.recv‚ 2.506311e+06$ disk./.usageGB‚248.3263053894043 disk.sda2.out‚0! disk.nvme0n1.in‚ 1187.62109375" disk.dm-0.out‚47354.6708984375 disk.drbd2266.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375) disk./.usagePercent‚16.66538671433131 disk.sda3.out‚ 321.15625) proc.memory.availableMB‚ 1620.46484375 disk.drbd1716.out‚0 disk.sda.in‚ 0.1953125 disk.sda.out‚ 321.15625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚47354.6708984375* proc.memory.percent‚17.162561416625977 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚47354.6708984375$ memory_percent‚90.10946750640869 proc.cpu.threads‚17 network.sent‚ 2.1413678e+07 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 321.15625 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.rbd1.out‚ 32.1240234375 cpu‚25.350170451695917 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 äø!2œ"- _wandbruntime‚6938  _runtime‚6938e¾Æ"À trainer/global_step‚8899 $ _timestamp‚1.788411835741711e+09 _runtime‚6938.729158309 _step‚729 & lr-AdamW/pg1‚5.435526019560587e-05 & lr-AdamW/pg2‚5.435526019560587e-05+–ñ©Ëž"Å & lr-AdamW/pg2‚5.435526019560587e-05 trainer/global_step‚8899 $ _timestamp‚1.788411835741711e+09 _runtime‚6938.729158309 _step‚729 & lr-AdamW/pg1‚5.435526019560587e-05Ùy“ak2Ÿ"- _wandbruntime‚6938  _runtime‚6938 Ž+ç' "áO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118154235 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504226993306624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525257345239552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7428909823541 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297600888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7457018671104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75569164 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884118357424457e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚511656574607281 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517801000933888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129515442 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7457018671104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7822244426240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚573237480052736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7429564261297 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129515442 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42585071 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚525258019604992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚517800982930944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7456362308608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565415418996224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504227010345984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42585521 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573237663422464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525257345239552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297580848 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129515444 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚525258019604992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118154687 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7822084996608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168085444 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 %’Èì'¡"æO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517801000933888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129515442 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7457018671104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7822244426240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚573237480052736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7429564261297 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129515442 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42585071 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚525258019604992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚517800982930944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7456362308608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565415418996224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504227010345984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42585521 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573237663422464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525257345239552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297580848 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129515444 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚525258019604992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118154687 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7822084996608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168085444 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118154235 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504226993306624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525257345239552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7428909823541 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297600888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7457018671104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75569164 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884118357424457e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚511656574607281 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088Úíüu2¢"- _wandbruntime‚6939  _runtime‚6939’cêËÖ!£"ëM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚517857335969792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7457181736448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118166956 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118167418 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚525315192084992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7457838112256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚517857353972736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75577334 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚504281914476544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297633384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511711623695689 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168103924 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚525315192084992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚517857335969792 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129529458 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚525314517706240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7429726258505 F@Devi‘¥–6ceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7457181736448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168083860 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚573299982485504 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297613318 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7823104496640 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42590084]éö&¤"ðM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511711623695689 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168103924 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚525315192084992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚517857335969792 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129529458 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚525314517706240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7429726258505 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7457181736448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168083860 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚573299982485504 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297613318 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7823104496640 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42590084 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚517857335969792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7457181736448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118166956 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118167418 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚525315192084992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7457838112256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚517857353972736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75577334 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚504281914476544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297633384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844ÛÏ–o2¥"- _wandbruntime‚6939  _runtime‚6939€FY¦"Ó % _timestamp‚1.7884118364434493e+09 0loss/FRt1/assignment_loss‚3.6086357793500348 0loss/FBt2/assignment_loss‚0.1795753219418905 .loss/FBt4/detection_loss‚2.410909342765808 &loss/total_loss‚40.762107907843486 /loss/FRt2/assignment_loss‚3.134998729012229 1loss/SRqqt1/assignment_loss‚1.635760652832687 0loss/FBt1/assignment_loss‚0.4297801402087013 /loss/FRt4/detection_loss‚2.5003716404340706 epoch‚7 /loss/FRt2/detection_loss‚2.0186924938209536 2loss/SRqqt3/assignment_loss‚0.8935026566112607 0loss/FBt4/assignment_loss‚0.6992528278729878 1loss/SRqqt3/detection_loss‚1.5276133209980767 _runtime‚6939.430858387 /loss/FRt3/detection_loss‚1.7928612249200804 0loss/SRqqt2/detection_loss‚1.796392059447814 1loss/SRqqt4/detection_loss‚1.7441622968073245 /loss/FBt1/detection_loss‚2.2426486909389496 /loss/FBt2/detection_loss‚2.4677811725573107 /loss/FRt3/assignment_loss‚1.152747917593571 1loss/SRqqt4/assignment_loss‚1.034980309954672 0loss/FRt4/assignment_loss‚2.1941893319694366 2loss/SRqqt2/assignment_loss‚1.1919906005567433 _step‚732 1loss/FBt3/assignment_loss‚0.08719977497383347 .loss/FRt1/detection_loss‚2.996998913826481 0loss/SRqqt1/detection_loss‚2.477111702784896 trainer/global_step‚8899 /loss/FBt3/detection_loss‚0.5439510056636778>a Œ^§"Ø /loss/FBt1/detection_loss‚2.2426486909389496 /loss/FBt2/detection_loss‚2.4677811725573107 /loss/FRt3/assignment_loss‚1.152747917593571 1loss/SRqqt4/assignment_loss‚1.034980309954672 0loss/FRt4/assignment_loss‚2.1941893319694366 2loss/SRqqt2/assignment_loss‚1.1919906005567433 _step‚732 1loss/FBt3/assignment_loss‚0.08719977497383347 .loss/FRt1/detection_loss‚2.996998913826481 0loss/SRqqt1/detection_loss‚2.477111702784896 trainer/global_step‚8899 /loss/FBt3/detection_loss‚0.5439510056636778 % _timestamp‚1.7884118364434493e+09 0loss/FRt1/assignment_loss‚3.6086357793500348 0loss/FBt2/assignment_loss‚0.1795753219418905 .loss/FBt4/detection_loss‚2.410909342765808 &loss/total_loss‚40.762107907843486 /loss/FRt2/assignment_loss‚3.134998729012229 1loss/SRqqt1/assignment_loss‚1.635760652832687 0loss/FBt1/assignment_loss‚0.4297801402087013 /loss/FRt4/detection_loss‚2.5003716404340706 epoch‚7 /loss/FRt2/detection_loss‚2.0186924938209536 2loss/SRqqt3/assignment_loss‚0.8935026566112607 0loss/FBt4/assignment_loss‚0.6992528278729878 1loss/SRqqt3/detection_loss‚1.5276133209980767 _runtime‚6939.430858387 /loss/FRt3/detection_loss‚1.7928612249200804 0loss/SRqqt2/detection_loss‚1.796392059447814 1loss/SRqqt4/detection_loss‚1.7441622968073245ÜüdXë’¨"‚(:‡ ÂÿãÔ…»œ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.724' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žx15w©"‚(:ì ÂÿãÔ„÷« " proc.memory.rssMB‚ 2811.9140625) disk./.usagePercent‚16.66769590229016# disk.nvme0c0n1.in‚ 1187.62109375( disk.nvme0c0n1.out‚47416.02294921875 disk.rbd1.in‚0* proc.memory.percent‚17.162561416625977 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚32.4375! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 321.890625 disk.drbd2265.out‚0 disk.sda3.out‚ 321.890625 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 network.sent‚ 2.1419679e+07 disk.drbd2266.in‚0 disk.drbd2080.out‚0$ memory_percent‚90.11118412017822) proc.memory.availableMB‚ 1620.18359375% disk./.usageGB‚248.36071395874023 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚47416.02294921875 cpu‚25.34949808306848 disk.sda1.out‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 network.recv‚ 2.510788e+06 disk.rbd0.in‚0# disk.dm-0.out‚47416.02294921875 disk.sda.out‚ 321.890625ø?žˆ’ª"‚(:‡ ÑÿãÔ®¼³ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.153' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÒî÷~«"‚(:ó ÑÿãÔÆ—¡  network.sent‚ 2.1528615e+07 disk.sda3.out‚ 322.73828125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1187.62109375 cpu‚25.348891353563467 disk.sda1.out‚0) disk./.usagePercent‚16.66807760952371 disk.sda.out‚ 322.73828125( disk.nvme0c0n1.out‚47487.11279296875) proc.memory.availableMB‚ 1620.17578125 disk.drbd1716.out‚0% disk./.usageGB‚248.36640167236328 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚32.4375 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚47487.11279296875 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 network.recv‚ 2.517073e+06 disk.dm-1.out‚ 322.73828125 disk.drbd2080.in‚0& disk.nvme0n1.out‚47487.11279296875$ memory_percent‚90.11123180389404" proc.memory.rssMB‚ 2811.9140625* proc.memory.percent‚17.162561416625977 proc.cpu.threads‚17î6c‚¬"‚(:÷ àÿãÔÔ°¨ & disk.nvme0n1.out‚47577.39990234375 disk.sda.out‚ 323.32421875) proc.memory.availableMB‚ 1620.16796875( disk.nvme0c0n1.out‚47577.39990234375 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 32.4697265625* disk./.usagePercent‚16.670473328029132 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2080.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2811.9140625 cpu‚25.348214774657762 disk.drbd1716.in‚0 network.recv‚ 2.52103e+06 disk.sda2.out‚0 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚47577.39990234375 disk.dm-1.out‚ 323.32421875 disk.rbd0.out‚0* proc.memory.percent‚17.162561416625977! disk.nvme0n1.in‚ 1187.62109375 disk.sda.in‚ 0.1953125$ memory_percent‚90.11127948760986 proc.cpu.threads‚17# disk./.usageGB‚248.402099609375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 network.sent‚ 2.1533937e+07 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 323.32421875ò’­"‚(:‡ àÿãÔΔ‘  gpu.0.gpu‚28.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eý(2®"- _wandbruntime‚6975  _runtime‚6975ƒ®Ïɯ"à ' lr-AdamW/pg1‚5.2782818416326385e-05 ' lr-AdamW/pg2‚5.2782818416326385e-05 trainer/global_step‚8949 % _timestamp‚1.7884118727646685e+09 _runtime‚6975.752188154 _step‚733EÏš}ΰ"È _step‚733 ' lr-AdamW/pg1‚5.2782818416326385e-05 ' lr-AdamW/pg2‚5.2782818416326385e-05 trainer/global_step‚8949 % _timestamp‚1.7884118727646685e+09 _runtime‚6975.752188154Ý›Ï 2±"- _wandbruntime‚6975  _runtime‚6975=ÏxLç'²"áO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506972216873984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚528115982987776 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130216244 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7469745162949 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576363202899968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚514441944997573 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7498004476928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7865075008000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42814115 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚520618652876288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7470399600705 _runtime‚6975.753187871 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299207700 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75977566 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8949 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7497348114432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42813669 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568497955879424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299227740 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7498004476928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚520618634873344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168991458 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168991458 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚514442616474689 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680)p€ ³"æO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168991458 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168991458 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚514442616474689 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚4ŽEHÛl48 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506972216873984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚528115982987776 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130216244 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7469745162949 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576363202899968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚514441944997573 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7498004476928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7865075008000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42814115 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚520618652876288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7470399600705 _runtime‚6975.753187871 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299207700 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75977566 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8949 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7497348114432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42813669 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568497955879424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299227740 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7498004476928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚520618634873344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944ÞwÄâ2´"- _wandbruntime‚6976  _runtime‚6976{Wë¸ð&µ"êM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240170 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520674987912192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚528173829833216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚520675005915136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169029976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576425522060288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507027121004544 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299260236 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75985734 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130230258 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169029976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75985732 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7498823918080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299240170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚514497665563097 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788411873464678e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118804317 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚528173155454464 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260236 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568559606617088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130230260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169009912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169009912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6976.452690854 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471216035877 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7498167542272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576425707513856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚528173829833216 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520675005915136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118803884 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0GEIõ&¶"ïM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130230260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169009912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169009912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6976.452690854 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471216035877 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7498167542272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576425707513856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚528173829833216 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520675005915136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118803884 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240170 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520674987912192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚528173829833216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚520675005915136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169029976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576425522060288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507027121004544 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299260236 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75985734 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130230258 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169029976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75985732 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7498823918080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299240170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚514497665563097 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788411873464678e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118804317 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚528173155454464 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260236 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568559606617088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ßç]¨2·"- _wandbruntime‚6976  _runtime‚6976ø¬[¸"Õ 0loss/SRqqt4/detection_loss‚1.663070927015864 %loss/total_loss‚42.96437727943446 % _timestamp‚1.7884118734669712e+09 0loss/FBt4/assignment_loss‚0.3255172900268614 1loss/SRqqt2/detection_loss‚1.9261758784030347 /loss/FBt1/detection_loss‚2.9621426105499267 .loss/FBt2/detection_loss‚2.533937088080815 .loss/FBt4/detection_loss‚2.580107460419337 trainer/global_step‚8949 0loss/FRt2/assignment_loss‚3.2471558216006255 /loss/FRt4/detection_loss‚2.2201219295360604 epoch‚7 _step‚736 2loss/SRqqt3/assignment_loss‚1.0696337931892939 1loss/SRqqt4/assignment_loss‚1.459517032910998 /loss/FRt1/detection_loss‚2.8989360723644495 /loss/FBt3/detection_loss‚0.6034886344937207 2loss/SRqqt2/assignment_loss‚1.3682412892936708 0loss/SRqqt3/detection_loss‚1.049475426164766 1loss/SRqqt1/detection_loss‚2.9584755275560464 _runtime‚6976.454352941 0loss/FRt1/assignment_loss‚3.4271086528897285 2loss/SRqqt1/assignment_loss‚1.5014509377272234 1loss/FBt3/assignment_loss‚0.17374269726814726 /loss/FRt2/detection_loss‚2.3210226124109226 /loss/FRt3/detection_loss‚1.5207430434520128 0loss/FRt4/assignment_loss‚2.5314890900436713 0loss/FBt1/assignment_loss‚0.2609213378280401 0loss/FRt3/assignment_loss‚1.4953668479059563 0loss/FBt2/assignment_loss‚0.8665352783032826×?g¿`¹"Ú 0loss/FBt4/assignment_loss‚0.3255172900268614 1loss/SRqqt2/detection_loss‚1.9261758784030347 /loss/FBt1/detection_loss‚2.9621426105499267 .loss/FBt2/detection_loss‚2.533937088080815 .loss/FBt4/detection_loss‚2.580107460419337 trainer/global_step‚8949 0loss/FRt2/assignment_loss‚3.2471558216006255 /loss/FRt4/detection_loss‚2.2201219295360604 epoch‚7 _step‚736 2loss/SRqqt3/assignment_loss‚1.0696337931892939 1loss/SRqqt4/assignment_loss‚1.459517032910998 /loss/FRt1/detection_loss‚2.8989360723644495 /loss/FBt3/detection_loss‚0.6034886344937207 2loss/SRqqt2/assignment_loss‚1.3682412892936708 0loss/SRqqt3/detection_loss‚1.049475426164766 1loss/SRqqt1/detection_loss‚2.9584755275560464 _runtime‚6976.454352941 0loss/FRt1/assignment_loss‚3.4271086528897285 2loss/SRqqt1/assignment_loss‚1.5014509377272234 1loss/FBt3/assignment_loss‚0.17374269726814726 /loss/FRt2/detection_loss‚2.3210226124109226 /loss/FRt3/detection_loss‚1.5207430434520128 0loss/FRt4/assignment_loss‚2.5314890900436713 0loss/FBt1/assignment_loss‚0.2609213378280401 0loss/FRt3/assignment_loss‚1.4953668479059563 0loss/FBt2/assignment_loss‚0.8665352783032826 0loss/SRqqt4/detection_loss‚1.663070927015864 %loss/total_loss‚42.96437727943446 % _timestamp‚1.7884118734669712e+09àªÂ?·’º"‚(:‡ ïÿãÔÜÜ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôT ƒ»"‚(:ø ïÿãÔÊÞƒ  disk.drbd2080.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚47665.67529296875 network.sent‚ 2.154011e+07) proc.memory.availableMB‚ 1620.13671875* disk./.usagePercent‚16.582883166271692 disk.sda3.out‚ 324.28515625 disk.drbd2266.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 disk.drbd2265.out‚0# disk.dm-0.out‚47665.67529296875 disk.sda.in‚ 0.1953125 disk.rbd1.in‚0 disk.dm-1.out‚ 324.28515625 disk.drbd2265.in‚0* proc.memory.percent‚17.162561416625977 cpu‚25.34757179707129 proc.cpu.threads‚17 network.recv‚ 2.525539e+06 disk.sda.out‚ 324.28515625 disk.sda3.in‚ 0.1953125 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1187.62109375% disk./.usageGB‚247.09694290161133 disk.sda1.in‚0 disk.rbd1.out‚ 32.4697265625 disk.dm-1.in‚ 0.1953125$ memory_percent‚90.11147022247314 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚47665.67529296875 disk.dm-0.in‚ 1187.62109375" proc.memory.rssMB‚ 2811.9140625ûU9ϼ"‚(:’ þÿãÔ±©© gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.355' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.270999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—YŠ‘½"‚(:ê þÿãԞˇ  disk.sda.out‚ 324.703125 disk.drbd2266.out‚0 disk.drbd2080.in‚0* proc.memory.percent‚17.162561416625977$ disk.nvme0n1.out‚47746.962890625 disk.sda3.out‚ 324.703125 disk.sda2.out‚0' proc.memory.availableMB‚ 1620.140625 disk.drbd1716.out‚0! disk.dm-0.out‚47746.962890625 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1187.62109375# disk.nvme0c0n1.in‚ 1187.62109375$ memory_percent‚90.11144638061523" proc.memory.rssMB‚ 2811.9140625 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2080.out‚0 disk.sda1.in‚0 network.recv‚ 2.531495e+06$ disk./.usageGB‚247.1170883178711 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 324.703125 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚47746.962890625 network.sent‚ 2.1648644e+07 disk.dm-0.in‚ 1187.62109375 disk.rbd0.in‚0 disk.sda€ãU=ä2.in‚0 cpu‚25.34694407013568* disk./.usagePercent‚16.584235142060013 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 32.7119140625 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125ðÜpí2¾"- _wandbruntime‚7012  _runtime‚7012ñ‡8Ç¿"Á & lr-AdamW/pg2‚5.122803772365854e-05 trainer/global_step‚8999 % _timestamp‚1.7884119096994827e+09 _runtime‚7012.686892266 _step‚737 & lr-AdamW/pg1‚5.122803772365854e-05qÑjÌÀ"Æ % _timestamp‚1.7884119096994827e+09 _runtime‚7012.686892266 _step‚737 & lr-AdamW/pg1‚5.122803772365854e-05 & lr-AdamW/pg2‚5.122803772365854e-05 trainer/global_step‚8999áäi +2Á"- _wandbruntime‚7012  _runtime‚7012ÌsÕ"æ'Â"àO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130917044 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579488562980352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7907902399488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130917044 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119427003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76385964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7538333920256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579488371222016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚8999 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509717406362624 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169917510 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530974620736000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7538990282752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523436304818688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚530974620736000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7538990282752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523436286815744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517227986864981 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169937548 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300834552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚530975295101440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130917042 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7511234940113 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚509717423401984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300854592 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119426551 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7538333920256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43040587 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚738 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43041037 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚571580492762624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚530975295101440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130917044 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119427003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76385964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7538333920256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579488371222016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚8999 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509717406362624 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169917510 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530974620736000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7538990282752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523436304818688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚530974620736000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7538990282752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523436286815744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517227986864981 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169937548 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300834552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚530975295101440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130917042 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7511234940113 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚509717423401984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300854592 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119426551 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7538333920256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43040587 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚738 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43041037 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚571580492762624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚530975295101440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509772310493184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300867022 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571642143500288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7908919909888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531031793202688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43045092 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7539153348096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169956028 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517284378907829 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119439224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7908743716352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509772327532544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76394132 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169935964 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571642119560192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7511396937321 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚579550863276544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130931060 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169935964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76394134 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517283707430505 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300887088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43045525 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚739 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130931058 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8999 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130931058 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7539809723904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300867022 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523492639854592 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119439659 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7512051375285 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523492657857536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚523492639854592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523492657857536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7539153348096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7539809723904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300867022 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523492639854592 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119439659 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7512051375285 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523492657857536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚523492639854592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523492657857536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7539153348096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509772310493184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300867022 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571642143500288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7908919909888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531031793202688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43045092 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7539153348096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169956028 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517284378907829 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119439224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7908743716352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509772327532544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064ãÜŒr¬2Ç"- _wandbruntime‚7013  _runtime‚70130½A#[È"Õ /loss/FBt2/detection_loss‚2.4752451926469803 trainer/global_step‚8999 _step‚740 0loss/FRt3/assignment_loss‚1.1756821186816107 .loss/FRt1/detection_loss‚3.262334880374727 _runtime‚7013.382791018 %loss/total_loss‚38.44265562811654 % _timestamp‚1.7884119103954082e+09 0loss/FBt1/assignment_loss‚0.1310176635493495 0loss/SRqqt2/detection_loss‚2.057531425599442 .loss/FBt1/detection_loss‚2.299869022199086 /loss/FRt2/assignment_loss‚2.184839252544486 2loss/SRqqt3/assignment_loss‚0.9097350969677791 2loss/SRqqt4/assignment_loss‚1.3603453159186185 1loss/FBt2/assignment_loss‚0.35558909829705954 /loss/FRt3/detection_loss‚1.5588686259799316 /loss/FRt4/detection_loss‚2.4874701913107526 epoch‚7 0loss/FRt1/assignment_loss‚2.9404261452811107 1loss/FBt3/assignment_loss‚0.07417857771137339 0loss/FRt4/assignment_loss‚2.3030550981109794 2loss/SRqqt2/assignment_loss‚0.8917229470558637 0loss/FBt4/assignment_loss‚0.0508136487333104 /loss/FRt2/detection_loss‚1.9891304160982477 0loss/SRqqt1/detection_loss‚2.478934446285511 .loss/FBt4/detection_loss‚2.508740097284317 2loss/SRqqt1/assignment_loss‚1.4683532596662128 1loss/SRqqt3/detection_loss‚1.0140725040691905 /loss/FBt3/detection_loss‚0.4861081635568246 1loss/SRqqt4/detection_loss‚1.9785924401937747¦µ`É"Ú epoch‚7 0loss/FRt1/assignment_loss‚2.9404261452811107 1loss/FBt3/assignment_loss‚0.07417857771137339 0loss/FRt4/assignment_loss‚2.3030550981109794 2loss/SRqqt2/assignment_loss‚0.8917229470558637 0loss/FBt4/assignment_loss‚0.0508136487333104 /loss/FRt2/detection_loss‚1.9891304160982477 0loss/SRqqt1/detection_loss‚2.478934446285511 .loss/FBt4/detection_loss‚2.508740097284317 2loss/SRqqt1/assignment_loss‚1.4683532596662128 1loss/SRqqt3/detection_loss‚1.0140725040691905 /loss/FBt3/detection_loss‚0.4861081635568246 1loss/SRqqt4/detection_loss‚1.9785924401937747 /loss/FBt2/detection_loss‚2.4752451926469803 trainer/global_step‚8999 _step‚740 0loss/FRt3/assignment_loss‚1.1756821186816107 .loss/FRt1/detection_loss‚3.262334880374727 _runtime‚7013.382791018 %loss/total_loss‚38.44265562811654 % _timestamp‚1.7884119103954082e+09 0loss/FBt1/assignment_loss‚0.1310176635493495 0loss/SRqqt2/detection_loss‚2.057531425599442 .loss/FBt1/detection_loss‚2.299869022199086 /loss/FRt2/assignment_loss‚2.184839252544486 2loss/SRqqt3/assignment_loss‚0.9097350969677791 2loss/SRqqt4/assignment_loss‚1.3603453159186185 1loss/FBt2/assignment_loss‚0.35558909829705954 /loss/FRt3/detection_loss‚1.5588686259799316 /loss/FRt4/detection_loss‚2.4874701913107526ä•¥¥’Ê"‚(:‡ €äÔíôµ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.136' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8272 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tg0uË"‚(:ê €äÔ¤ì˜  disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375" proc.memory.rssMB‚ 2811.9140625* proc.memory.percent‚17.162561416625977 cpu‚25.346310823436102 disk.dm-1.out‚ 325.546875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚47820.279296875) proc.memory.availableMB‚ 1620.07421875* disk./.usagePercent‚16.585536172349023# disk./.usageGB‚247.136474609375 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 325.546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.sda.out‚ 325.546875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0& disk.nvme0c0n1.out‚47820.279296875 disk.dm-0.in‚ 1187.62109375! disk.dm-0.out‚47820.279296875 network.sent‚ 2.1654997e+07 disk.drbd1716.out‚0# memory_percent‚90.1118516921997 disk.rbd1.out‚ 32.7119140625 network.recv‚ 2.53627e+06 disk.drbd2266.in‚0 disk.drbd1716.in‚0ÀLö’Ì"‚(:‡ œ€äÔÛÛû gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ýñ‚Í"‚(:÷ œ€äÔžÊÿ disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 326.23046875* proc.memory.percent‚17.162561416625977 cpu‚25.345718128646126 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1187.62109375 network.sent‚ 2.1762975e+07 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0# disk.nvme0c0n1.in‚ 1187.62109375( disk.nvme0c0n1.out‚47929.41845703125 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0# disk.dm-0.out‚47929.41845703125( proc.memory.availableMB‚ 1620.0703125 disk.rbd0.in‚0& disk.nvme0n1.out‚47929.41845703125$ memory_percent‚90.11187553405762 disk.sda3.out‚ 326.23046875 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 32.744140625 disk.dm-0.in‚ 1187.62109375) disk./.usagePercent‚16.58729110399624% disk./.usageGB‚247.16262435913086 disk.sda2.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda.out‚ 326.23046875 disk.drbd2266.in‚0 disk.drbd2265.in‚0 network.recv‚ 2.541844e+06 disk.sda2.out‚0" proc.memory.rssMB‚ 2811.9140625‘ôYÅ2Î"- _wandbruntime‚7049  _runtime‚7049»"€ÇÏ"Á & lr-AdamW/pg1‚4.969124213156823e-05 & lr-AdamW/pg2‚4.969124213156823e-05 trainer/global_step‚9049 % _timestamp‚1.7884119467926705e+09 _runtime‚7049.780098189 _step‚741óãØÌÐ"Æ % _timestamp‚1.7884119467926705e+09 _runtime‚7049.780098189 _step‚741 & lr-AdamW/pg1‚4.969124213156823e-05 & lr-AdamW/pg2‚4.969124213156823e-05 trainer/global_step‚9049å „kæ2Ñ"- _wandbruntime‚7049  _runtime‚7049pƒ“ç'Ò"áO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520014700209505 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚526253938758144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170863600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76794366 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582614084736512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7951055090688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884119467933688e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7579319726080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7579976088576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7579976088576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533833932849664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7552070279521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170843562 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131617844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131617842 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43265246 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚526253956761088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582613903463936 _runtime‚7049.781044935 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302481444 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170843562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43265685 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533833258484224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚512462612890624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526253938758144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120059610 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120060051 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7551415841765 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170863600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7950897758208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131617844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76794364 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302461404 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚9049 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚526253956761088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574663029645824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚574663005705728ó:­Ó"æO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚526253938758144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170863600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76794366 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582614084736512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7951055090688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884119467933688e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7579319726080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7579976088576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7579976088576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533833932849664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7552070279521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170843562 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131617844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131617842 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43265246 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚526253956761088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582613903463936 _runtime‚7049.781044935 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302481444 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170843562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43265685 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533833258484224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚512462612890624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526253938758144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120059610 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120060051 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7551415841765 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170863600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7950897758208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131617844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76794364 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302461404 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚9049 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚526253956761088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574663029645824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚574663005705728 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520014700209505æ•â†’Ô"‚(:‡ «€äÔÓê” gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04CÈŸ‚Õ"‚(:÷ «€äÔç“  disk.rbd1.out‚ 32.744140625& disk.nvme0n1.out‚48008.74658203125 disk.dm-1.out‚ 326.56640625 disk.drbd2080.out‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2811.9140625 cpu‚25.3450546714051 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 326.56640625 disk.drbd2266.out‚0 network.sent‚ 2.1768974e+07 disk.drbd1716.out‚0* proc.memory.percent‚17.162561416625977 disk.sda2.in‚0 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 326.56640625* disk./.usagePercent‚16.588339710856488% disk./.usageGB‚247.17824935913086# disk.dm-0.out‚48008.74658203125 network.recv‚ 2.546134e+06# disk.nvme0c0n1.in‚ 1187.62109375! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.drbd2080.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚48008.74658203125 disk.drbd1716.in‚0 disk.sda3.in‚ 0.1953125) proc.memory.availableMB‚ 1620.33203125 disk.drbd2266.in‚0$ memory_percent‚90.11027812957764 proc.cpu.threads‚17 disk.sda1.out‚0}¢ û2Ö"- _wandbruntime‚7050  _runtime‚7050Ýè[ñ&×"ëM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43270101 % _timestamp‚1.7884119474861639e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582676601677824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚7050.474399192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76802534 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚512517534060544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7951757683712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚533891105329664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131631860 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574724656443392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131631858 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚533891105329664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚9049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302493874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7580139153920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚526310309799936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7580139153920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302513940 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120072191 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7580795529728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302493874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7552232276729 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526310291796992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7580795529728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513940 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170882080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170882080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170862016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚512517517021184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120072635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76802532ªË1ö&Ø"ðM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120072191 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7580795529728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302493874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7552232276729 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526310291796992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7580795529728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513940 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170882080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170882080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170862016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚512517517021184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120072635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76802532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43270101 % _timestamp‚1.7884119474861639e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582676601677824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚7050.474399192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76802534 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚512517534060544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7951757683712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚533891105329664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131631860 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574724656443392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131631858 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚533891105329664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚9049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302493874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7580139153920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚526310309799936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7580139153920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302513940 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444ç˜-@2Ù"- _wandbruntime‚7050  _runtime‚7050ôAãUÚ"Ï -loss/FRt4/assignment_loss‚2.3874669795235 0loss/FBt1/assignment_loss‚0.7903294950723648 1loss/FBt2/assignment_loss‚0.18128841536163165 .loss/FRt2/detection_loss‚2.115058007101871 1loss/SRqqt3/detection_loss‚1.0699275962801442 2loss/SRqqt4/assignment_loss‚1.3144463224957386 0loss/FBt4/assignment_loss‚0.3870812389868661 /loss/FRt4/detection_loss‚2.2871057614684105 1loss/SRqqt2/detection_loss‚1.7674483520910145 /loss/FBt1/detection_loss‚2.6537421345710754 /loss/FBt2/detection_loss‚2.3838253796100615 /loss/FRt2/assignment_loss‚3.081196358037549 0loss/FBt3/assignment_loss‚0.2796108991522186 0loss/SRqqt1/detection_loss‚2.502173250541091 /loss/FBt3/detection_loss‚0.6561373746297203 trainer/global_step‚9049 % _timestamp‚1.7884119474884374e+09 _runtime‚7050.475838387 /loss/FRt3/detection_loss‚1.4865330373173329 .loss/FBt4/detection_loss‚4.091306686401367 _step‚744 1loss/SRqqt4/detection_loss‚1.4466519967963298 $loss/total_loss‚41.9833306955818 2loss/SRqqt1/assignment_loss‚1.5122372088953853 0loss/FRt3/assignment_loss‚0.9841259486639677 2loss/SRqqt2/assignment_loss‚1.2752630779519678 epoch‚7 0loss/FRt1/assignment_loss‚3.2915457050005594 0loss/SRqqt3/assignment_loss‚1.12855833366565 .loss/FRt1/detection_loss‚2.910271135965983d§adZÛ"Ô 2loss/SRqqt1/assignment_loss‚1.5122372088953853 0loss/FRt3/assignment_loss‚0.9841259486639677 2loss/SRqqt2/assignment_loss‚1.2752630779519678 epoch‚7 0loss/FRt1/assignment_loss‚3.2915457050005594 0loss/SRqqt3/assignment_loss‚1.12855833366565 .loss/FRt1/detection_loss‚2.910271135965983 -loss/FRt4/assignment_loss‚2.3874669795235 0loss/FBt1/assignment_loss‚0.7903294950723648 1loss/FBt2/assignment_loss‚0.18128841536163165 .loss/FRt2/detection_loss‚2.115058007101871 1loss/SRqqt3/detection_loss‚1.0699275962801442 2loss/SRqqt4/assignment_loss‚1.3144463224957386 0loss/FBt4/assignment_loss‚0.3870812389868661 /loss/FRt4/detection_loss‚2.2871057614684105 1loss/SRqqt2/detection_loss‚1.7674483520910145 /loss/FBt1/detection_loss‚2.6537421345710754 /loss/FBt2/detection_loss‚2.3838253796100615 /loss/FRt2/assignment_loss‚3.081196358037549 0loss/FBt3/assignment_loss‚0.2796108991522186 0loss/SRqqt1/detection_loss‚2.502173250541091 /loss/FBt3/detection_loss‚0.6561373746297203 trainer/global_step‚9049 % _timestamp‚1.7884119474884374e+09 _runtime‚7050.475838387 /loss/FRt3/detection_loss‚1.4865330373173329 .loss/FBt4/detection_loss‚4.091306686401367 _step‚744 1loss/SRqqt4/detection_loss‚1.4466519967963298 $loss/total_loss‚41.9833306955818è :ÁÉ‘Ü"‚(:† º€äÔ¾Ôõ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.411 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®Ä‘À|Ý"‚(:ñ º€äԛĤ % disk./.usageGB‚247.19575119018555 disk.rbd1.out‚ 32.900390625 disk.sda.out‚ 327.0703125 network.recv‚ 2.551646e+06' disk.nvme0c0n1.out‚48071.6103515625 disk.rbd1.in‚0" disk.dm-0.out‚48071.6103515625 disk.dm-1.in‚ 0.1953125 cpu‚25.344429437717956 proc.cpu.threads‚17* disk./.usagePercent‚16.589514273423582 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0! disk.nvme0n1.in‚ 1187.62109375% disk.nvme0n1.out‚48071.6103515625 disk.drbd2265.out‚0 network.sent‚ 2.1865293e+07 disk.drbd2265.in‚0( proc.memory.availableMB‚ 1620.0390625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 327.0703125 disk.sda3.in‚ 0.1953125# memory_percent‚90.1120662689209 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2811.9140625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 327.0703125* proc.memory.percent‚17.162561416625977 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125H]ª’Þ"‚(:‡ É€äÔœŽƒ gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u6‘xß"‚(:í É€äÔ»þæ! disk.nvme0n1.in‚ 1187.62109375 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 cpu‚25.3437762681303 disk.sda3.out‚ 327.640625 disk.sda.out‚ 327.640625$ memory_percent‚90.11359214782715% disk./.usageGB‚247.21454620361328 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0" disk.dm-0.out‚48147.5732421875 disk.sda.in‚ 0.1953125 proc.cpu.threads‚17' disk.nvme0c0n1.out‚48147.5732421875 disk.rbd1.out‚ 32.900390625 disk.dm-0.in‚ 1187.62109375* disk./.usagePercent‚16.590775622544783 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 network.recv‚ 2.557683e+06 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.dm-1.in‚ 0.1953125 disk.sda2.out‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 1619.7890625" proc.memory.rssMB‚ 2811.9140625 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0* proc.memory.percent‚17.162561416625977 network.sent‚ 2.1963772e+07 disk.sda1.out‚0% disk.nvme0n1.out‚48147.5732421875 disk.dm-1.out‚ 327.640625­æl2à"- _wandbruntime‚7086  _runtime‚7086ˆ¦}Æá"À $ _timestamp‚1.788411983361075e+09 _runtime‚7086.348479848 _step‚745 & lr-AdamW/pg1‚4.817275190595349e-05 & lr-AdamW/pg2‚4.817275190595349e-05 trainer/global_step‚9099pü¿Ëâ"Å & lr-AdamW/pg1‚4.817275190595349e-05 & lr-AdamW/pg2‚4.817275190595349e-05 trainer/global_step‚9099 $ _timestamp‚1.788411983361075e+09 _runtime‚7086.348479848 _step‚745éÙ‘…&2ã"- _wandbruntime‚7086  _runtime‚7086âÉDi)ä"àO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚536692570597888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77202766 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚515207819418624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592905618929 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536691896232448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171789652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43492800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚746 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚529071590700544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171769614 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529071608703488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactitý1g½ve_split_bytes.small_pool.peak‚ 268325376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132318642 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529071590700544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77202764 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7620305531904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚585739634017792 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171769614 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522800070599797 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529071608703488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7620961894400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132318644 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7592251181173 $ _timestamp‚1.788411983361838e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132318644 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7620305531904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚585739442259456Ò¤Þë'å"åO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132318642 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529071590700544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77202764 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7620305531904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚585739634017792 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171769614 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522800070599797 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529071608703488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7620961894400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132318644 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7592251181173 $ _timestamp‚1.788411983361838e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132318644 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7620305531904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚585739442259456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚536692570597888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77202766 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚515207819418624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592905618929 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536691896232448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171789652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43492800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚746 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚529071590700544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171769614 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529071608703488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414ê•Në2æ"- _wandbruntime‚7087  _runtime‚7087²‹µñ&ç"ëM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621781335552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚515262740588544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚585801958141440 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7593067616137 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚7087.040080942 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529127961742336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43497308 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43496860 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132332660 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522856462642645 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529127943739392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7621781335552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚747 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304140792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132332660 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132332658 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120707792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚536749068699136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171788068 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304120726 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529127961742336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7593722054101 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585802143595008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577807193326592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304120726 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171788068 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872ß-¶ìè"ðM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522856462642645 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529127943739392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7621781335552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚747 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304140792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132332660 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132332658 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120707792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚536749068699136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171788068 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304120726 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529127961742336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7593722054101 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585802143595008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577807193326592 OADeviceStatsMonitor.o ?öüän_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304120726 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171788068 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621781335552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚515262740588544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚585801958141440 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7593067616137 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚7087.040080942 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529127961742336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43497308 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43496860 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132332660 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928ëp¯ÃP2é"- _wandbruntime‚7087  _runtime‚70878Ç·ì]ê"× /loss/FRt2/detection_loss‚2.4988863278775875 /loss/SRqqt3/detection_loss‚1.04432523516672 /loss/FBt2/detection_loss‚2.2360992630322776 % _timestamp‚1.7884119840538485e+09 1loss/FBt3/assignment_loss‚0.16222698390267629 /loss/FRt3/detection_loss‚1.8209450602117512 /loss/FBt1/detection_loss‚2.7553201743534634 epoch‚7 trainer/global_step‚9099 0loss/FRt3/assignment_loss‚1.6063500046730042 2loss/SRqqt4/assignment_loss‚1.4872088938675545 1loss/FBt2/assignment_loss‚0.12705980493531874 .loss/FRt1/detection_loss‚3.123046221832434 0loss/SRqqt1/detection_loss‚2.592708687697138 /loss/FBt4/detection_loss‚2.8210251927375793 /loss/FRt1/assignment_loss‚2.789220387736956 1loss/SRqqt2/assignment_loss‚1.524761895124208 1loss/FBt1/assignment_loss‚0.35013698315846603 /loss/FBt3/detection_loss‚0.4725396309884011 2loss/SRqqt1/assignment_loss‚1.8324932711465018 /loss/FRt4/detection_loss‚2.2743432405202286 _runtime‚7087.041235671 _step‚748 2loss/SRqqt3/assignment_loss‚0.9853386394383888 1loss/SRqqt2/detection_loss‚1.6771480969407342 &loss/total_loss‚41.928319762549364 0loss/FRt4/assignment_loss‚2.6238560870937677 1loss/SRqqt4/detection_loss‚1.6864504874856383 0loss/FRt2/assignment_loss‚3.0456720656818814 0loss/FBt4/assignment_loss‚0.3911571269466852ćbë"Ü trainer/global_step‚9099 0loss/FRt3/assignment_loss‚1.6063500046730042 2loss/SRqqt4/assignment_loss‚1.4872088938675545 1loss/FBt2/assignment_loss‚0.12705980493531874 .loss/FRt1/detection_loss‚3.123046221832434 0loss/SRqqt1/detection_loss‚2.592708687697138 /loss/FBt4/detection_loss‚2.8210251927375793 /loss/FRt1/assignment_loss‚2.789220387736956 1loss/SRqqt2/assignment_loss‚1.524761895124208 1loss/FBt1/assignment_loss‚0.35013698315846603 /loss/FBt3/detection_loss‚0.4725396309884011 2loss/SRqqt1/assignment_loss‚1.8324932711465018 /loss/FRt4/detection_loss‚2.2743432405202286 _runtime‚7087.041235671 _step‚748 2loss/SRqqt3/assignment_loss‚0.9853386394383888 1loss/SRqqt2/detection_loss‚1.6771480969407342 &loss/total_loss‚41.928319762549364 0loss/FRt4/assignment_loss‚2.6238560870937677 1loss/SRqqt4/detection_loss‚1.6864504874856383 0loss/FRt2/assignment_loss‚3.0456720656818814 0loss/FBt4/assignment_loss‚0.3911571269466852 /loss/FRt2/detection_loss‚2.4988863278775875 /loss/SRqqt3/detection_loss‚1.04432523516672 /loss/FBt2/detection_loss‚2.2360992630322776 % _timestamp‚1.7884119840538485e+09 1loss/FBt3/assignment_loss‚0.16222698390267629 /loss/FRt3/detection_loss‚1.8209450602117512 /loss/FBt1/detection_loss‚2.7553201743534634 epoch‚7ì­]½å’ì"‚(:‡ Ø€äÔ¨‚® gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.633' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F’Ð~í"‚(:ó Ø€äÔ‚¿ž ' disk.nvme0c0n1.out‚48245.6357421875 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125% disk.nvme0n1.out‚48245.6357421875 network.sent‚ 2.1969055e+07 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2265.in‚0$ memory_percent‚89.91713523864746 proc.cpu.threads‚17 disk.sda.out‚ 328.21484375 disk.rbd1.in‚0( proc.memory.availableMB‚ 1651.9765625 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0) disk./.usagePercent‚16.59149858782148 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 328.21484375 cpu‚25.34318187984707 network.recv‚ 2.561551e+06$ disk./.usageGB‚247.2253189086914 disk.sda3.out‚ 328.21484375 disk.drbd2265.out‚0* proc.memory.percent‚17.162561416625977 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.rbd1.out‚ 32.9326171875! disk.nvme0n1.in‚ 1187.62109375" disk.dm-0.out‚48245.6357421875" proc.memory.rssMB‚ 2811.9140625 disk.drbd2266.in‚0 disk.sda2.in‚0ä$dWœî"‚(:‘ ç€äÔù«¤ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.91' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.382000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ç¥ãzï"‚(:ï ç€äÔŸ«‡  disk.sda1.in‚0' disk.nvme0c0n1.out‚48323.8740234375 disk.drbd1716.out‚0 disk.rbd1.out‚ 32.9326171875! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2266.in‚0* disk./.usagePercent‚16.593304976983077 disk.dm-1.out‚ 328.640625 disk.drbd2265.in‚0 disk.drbd2080.in‚0' proc.memory.availableMB‚ 1754.734375 disk.sda.out‚ 328.640625* proc.memory.percent‚17.162561416625977 proc.cpu.threads‚17 network.recv‚ 2.567661e+06 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 network.sent‚ 2.2077784e+07 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0 disk.dm-0.in‚ 1187.62109375% disk./.usageGB‚247.25223541259766 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375" proc.memory.rssMB‚ 2811.9140625 disk.dm-1.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚48323.8740234375" disk.dm-0.out‚48323.8740234375 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 328.640625$ memory_percent‚89.28995132446289 cpu‚25.342524447704847 disk.drbd2266.out‚06ðp‚2ð"- _wandbruntime‚7123  _runtime‚7123zcÂÇñ"Á & lr-AdamW/pg1‚4.667288349790143e-05 & lr-AdamW/pg2‚4.667288349790143e-05 trainer/global_step‚9149 % _timestamp‚1.7884120207018952e+09 _runtime‚7123.689318737 _step‚749ó ¾HÌò"Æ _runtime‚7123.689318737 _step‚749 & lr-AdamW/pg1‚4.667288349790143e-05 & lr-AdamW/pg2‚4.667288349790143e-05 trainer/global_step‚9149 % _timestamp‚1.7884120207018952e+09íB‡éÈ2ó"- _wandbruntime‚7123  _runtime‚7123íƇç'ô"áO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚539550533980672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚539551208346112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580828079472128 trainer/global_step‚9149 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172695666 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77611164 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133019442 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525586783944321 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172715704 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133019444 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7661947700224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8036785395712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305735148 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588865050334720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305715108 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531889242642944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172695666 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531889260645888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172715704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588864864867840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8036946922496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531889260645888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43720662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚539551208346112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133019442 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7633086520581 _step‚750 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531889242642944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539550533980672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121331828 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517953042985984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531889260645888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43720662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚539551208346112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133019442 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7633086520581 _step‚750 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531889242642944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539550533980672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121331828 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517953042985984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚539551208346112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580828079472128 trainer/global_step‚9149 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172695666 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77611164 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133019442 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525586783944321 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172715704 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133019444 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7661947700224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8036785395712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305735148 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588865050334720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305715108 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531889242642944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172695666 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531889260645888îî-bh2ö"- _wandbruntime‚7124  _runtime‚7124á2‡™÷"êM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121344029 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518007947116544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚518007930077184 % _timestamp‚1.7884120213904076e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7662767141376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚539608380826112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚525641833032729 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531945595681792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531945595681792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525642504510053 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133033458 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43725146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8037624299008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172734184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767644 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539608380826112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172734184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531945613684736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7662110765568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77619334 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚ 7124.37831992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7662767141376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580889730209792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43724697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7634557393509 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305767644 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944ÒDc õ&ø"ïM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77619332 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588927354508800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚588927552545280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8037798395392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121344480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7633902955545 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133033458 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539608380826112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172734184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531945613684736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7662110765568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77619334 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚ 7124.37831992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7662767141376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580889730209792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43724697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7634557393509 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305767644 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121344029 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518007947116544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚518007930077184 % _timestamp‚1.7884120213904076e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7662767141376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚539608380826112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚525641833032729 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531945595681792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531945595681792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525642504510053 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133033458 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43725146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8037624299008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172734184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767644 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542409171728896 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121966984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520698249513984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚534706894585344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7674576297745 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133720244 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173641756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43946968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591990370102784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520698232474624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307341960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133720242 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚7160.740999185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 % _timestamp‚1.7884120577533057e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173621718 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583910616355328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78019564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133720242 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7702933506048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7673921859989 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7702277143552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7702277143552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.s+eyðÂmall_pool.allocated‚ 7702933506048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133720244 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8079753747456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542409846094336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534706912588288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚583910640295424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173621718 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528372825811729 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121966532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚534706894585344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚542409846094336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362000 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528372154334613 _step‚754 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121966532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚534706894585344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚542409846094336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362000 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528372154334613 _step‚754 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542409171728896 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121966984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520698249513984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚534706894585344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7674576297745 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133720244 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173641756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43946968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591990370102784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520698232474624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307341960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133720242 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚7160.740999185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 % _timestamp‚1.7884120577533057e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173621718 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583910616355328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78019564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133720242 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7702933506048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7673921859989 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7702277143552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7702277143552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7702933506048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133720244 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8079753747456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542409846094336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534706912588288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚583910640295424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944ò¾®12ˆ#- _wandbruntime‚7161  _runtime‚7161LhÄ]ñ&‰#ëM PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121979623 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78027734 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7703096571392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121979162 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173660236 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534763247624192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7703096571392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7675392732917 _step‚755 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133734258 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43951430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583972291033088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7703752947200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78027732 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7674738294953 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7703752947200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534763265627136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚534763265627136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542466344195584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592053076813312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚583972267092992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133734260 •ŸP Š#ðM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78027732 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7674738294953 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7703752947200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534763265627136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚534763265627136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542466344195584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592053076813312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚583972267092992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133734260 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121979623 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78027734 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7703096571392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121979162 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173660236 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534763247624192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7703096571392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7675392732917 _step‚755 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133734258 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43951430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583972291033088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7703752947200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952ónÉ7{2‹#- _wandbruntime‚7161  _runtime‚7161¼å]Œ#× 2loss/SRqqt2/assignment_loss‚1.4998851316825796 /loss/FBt3/detection_loss‚0.5025796632996246 _runtime‚7161.433007828 1loss/SRqqt4/detection_loss‚1.7197156694589877 _step‚756 trainer/global_step‚9199 .loss/FRt2/detection_loss‚2.086533978388731 0loss/FRt2/assignment_loss‚3.2294375626546032 0loss/FBt1/assignment_loss‚0.4341930040682165 1loss/FBt4/assignment_loss‚0.45547645854954183 /loss/FBt2/detection_loss‚2.5767842910506507 &loss/total_loss‚40.908492044662026 0loss/FRt1/assignment_loss‚3.1631090994234436 1loss/SRqqt1/detection_loss‚2.1669117995670866 epoch‚7 $ _timestamp‚1.788412058445586e+09 0loss/FBt2/assignment_loss‚0.4099970116089496 1loss/SRqqt2/detection_loss‚1.7183185705604653 0loss/SRqqt3/detection_loss‚0.885083451175264 /loss/FBt4/detection_loss‚1.9467335885221309 0loss/FRt3/assignment_loss‚1.5041918883136673 0loss/FRt4/assignment_loss‚2.7914589762687685 2loss/SRqqt1/assignment_loss‚1.7759222294603074 2loss/SRqqt3/assignment_loss‚0.9140343326410013 .loss/FRt1/detection_loss‚2.960324815026036 .loss/FRt4/detection_loss‚2.397534594270918 2loss/SRqqt4/assignment_loss‚1.4519003705651152 1loss/FBt3/assignment_loss‚0.11556673502863263 /loss/FRt3/detection_loss‚1.6351784935184555 /loss/FBt1/detection_loss‚2.5676203295588493J\'b#Ü 0loss/FRt3/assignment_loss‚1.5041918883136673 0loss/FRt4/assignment_loss‚2.7914589762687685 2loss/SRqqt1/assignment_loss‚1.7759222294603074 2loss/SRqqt3/assignment_loss‚0.9140343326410013 .loss/FRt1/detection_loss‚2.960324815026036 .loss/FRt4/detection_loss‚2.397534594270918 2loss/SRqqt4/assignment_loss‚1.4519003705651152 1loss/FBt3/assignment_loss‚0.11556673502863263 /loss/FRt3/detection_loss‚1.6351784935184555 /loss/FBt1/detection_loss‚2.5676203295588493 2loss/SRqqt2/assignment_loss‚1.4998851316825796 /loss/FBt3/detection_loss‚0.5025796632996246 _runtime‚7161.433007828 1loss/SRqqt4/detection_loss‚1.7197156694589877 _step‚756 trainer/global_step‚9199 .loss/FRt2/detection_loss‚2.086533978388731 0loss/FRt2/assignment_loss‚3.2294375626546032 0loss/FBt1/assignment_loss‚0.4341930040682165 1loss/FBt4/assignment_loss‚0.45547645854954183 /loss/FBt2/detection_loss‚2.5767842910506507 &loss/total_loss‚40.908492044662026 0loss/FRt1/assignment_loss‚3.1631090994234436 1loss/SRqqt1/detection_loss‚2.1669117995670866 epoch‚7 $ _timestamp‚1.788412058445586e+09 0loss/FBt2/assignment_loss‚0.4099970116089496 1loss/SRqqt2/detection_loss‚1.7183185705604653 0loss/SRqqt3/detection_loss‚0.885083451175264 /loss/FBt4/detection_loss‚1.9467335885221309ôŽfEx’Ž#‚(:‡ £äÔØ„ò gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AW¬#‚(:ö £äÔº·‡  disk.sda1.out‚0( disk.nvme0c0n1.out‚48619.83642578125! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.sda.out‚ 330.86328125 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚48619.83642578125 disk.drbd2265.out‚0 network.recv‚ 2.586226e+06) disk./.usagePercent‚16.59666200377957$ memory_percent‚88.68658542633057) proc.memory.availableMB‚ 1853.58984375 disk.drbd2266.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.162561416625977# disk.dm-0.out‚48619.83642578125 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 330.86328125$ disk./.usageGB‚247.3022575378418 disk.sda3.out‚ 330.86328125 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 disk.drbd2265.in‚0 cpu‚25.340057632857423 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 33.20703125 disk.sda.in‚ 0.1953125 proc.cpu.threads‚17 network.sent‚ 2.2204297e+07 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2811.9140625 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0NFn¾’#‚(:‡ ²äÔÿ²ý gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£}µx‘#‚(:í ²äÔµœå cpu‚25.339428739034098$ disk.nvme0n1.out‚48703.923828125 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 331.3671875 disk.drbd2266.in‚0 disk.drbd2080.out‚0$ memory_percent‚88.13729286193848( proc.memory.availableMB‚ 1943.5859375) disk./.usagePercent‚16.59768040175078 disk.dm-1.out‚ 331.3671875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 331.3671875 network.recv‚ 2.592978e+06& disk.nvme0c0n1.out‚48703.923828125% disk./.usageGB‚247.31743240356445# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚ 33.36328125! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda1.out‚0 network.sent‚ 2.2313264e+07 disk.drbd2080.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.162561416625977 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0! disk.dm-0.out‚48703.923828125 disk.sda3.in‚ 0.1953125" proc.memory.rssMB‚ 2811.9140625üúk2’#- _wandbruntime‚7197  _runtime‚7197c€›É“#à ' lr-AdamW/pg1‚4.3730258469899756e-05 ' lr-AdamW/pg2‚4.3730258469899756e-05 trainer/global_step‚9249 % _timestamp‚1.7884120948721175e+09 _runtime‚7197.859561278 _step‚757öCýcΔ#È ' lr-AdamW/pg1‚4.3730258469899756e-05 ' lr-AdamW/pg2‚4.3730258469899756e-05 trainer/global_step‚9249 % _timestamp‚1.7884120948721175e+09 _runtime‚7197.859561278 _step‚757õä=q2•#- _wandbruntime‚7197  _runtime‚7197KƒBç'–#áO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595116018194944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44175536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545267809477120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134421042 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531158196202021 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134421044 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚537524564530688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8122841016320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174547770 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44175080 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7715411637153 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7714757199397 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78427964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531158867679137 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545268483842560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚537524564530688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8122683683840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78427966 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523443439002624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545267809477120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚537524546527744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844Jyomì'—#æO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134421042 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531158196202021 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134421044 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚537524564530688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8122841016320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174547770 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44175080 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7715411637153 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7714757199397 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78427964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531158867679137 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545268483842560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚537524564530688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8122683683840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78427966 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523443439002624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545267809477120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚537524546527744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595116018194944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44175536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545267809477120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0öÖ¤üã2˜#- _wandbruntime‚7198  _runtime‚7198·:`½˜ ™#ëM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122616153 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7198.551959617 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309001282 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚523498343133184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134435058 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚537580899566592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78436134 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587054803976192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134435060 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134435060 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174586288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884120955640206e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309021348 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134435058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44179569 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMon„u°_Yitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587054827916288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚9249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8123552560640 _step‚759 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚545325656322560 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚537580917569536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7715573634361 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7744738753024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545324981943808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595178537796096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8123709879808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7744738753024 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174586288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122615701 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414sÂŒ+ö&š#ðM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309021348 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134435058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44179569 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587054827916288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚9249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8123552560640 _step‚759 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚545325656322560 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚537580917569536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7715573634361 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7744738753024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545324981943808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595178537796096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8123709879808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7744738753024 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174586288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122615701 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122616153 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7198.551959617 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309001282 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚523498343133184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134435058 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚537580899566592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78436134 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587054803976192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134435060 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134435060 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174586288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884120955640206e+09÷¢Óe©2›#- _wandbruntime‚7198  _runtime‚7198ˆ@i_œ#Ù 1loss/FBt3/assignment_loss‚0.17463196535148087 .loss/FRt2/detection_loss‚2.130402731276523 /loss/FRt3/detection_loss‚1.7923457155173474 /loss/FRt4/detection_loss‚2.2835514307022096 /loss/FBt1/detection_loss‚2.2318234627063456 _runtime‚7198.553151496 /loss/FRt1/detection_loss‚3.0869506412538987 epoch‚7 trainer/global_step‚9249 0loss/FBt2/assignment_loss‚0.4514360936232565 2loss/SRqqt2/assignment_loss‚0.8636722262948752 1loss/FBt4/assignment_loss‚0.49581297375594396 /loss/FRt2/assignment_loss‚2.435032943073584 2loss/SRqqt4/assignment_loss‚1.4891204691313682 1loss/SRqqt3/detection_loss‚0.8958417960142685 2loss/SRqqt1/assignment_loss‚1.3735894995076317 1loss/FBt1/assignment_loss‚0.18340890942356333 0loss/SRqqt1/detection_loss‚2.231374795096261 1loss/SRqqt2/detection_loss‚1.8347955100238322 1loss/SRqqt4/detection_loss‚2.0681108962348165 /loss/FBt2/detection_loss‚2.4774498397653755 0loss/FBt3/detection_loss‚0.40888829620719963 % _timestamp‚1.7884120955657785e+09 /loss/FRt1/assignment_loss‚3.176505002482184 0loss/FRt4/assignment_loss‚2.8055119693279265 /loss/FBt4/detection_loss‚2.6292543496404375 0loss/FRt3/assignment_loss‚0.9410068446939642 2loss/SRqqt3/assignment_loss‚1.0415769995079707 %loss/total_loss‚39.50209536061226 _step‚760 À>^d#Þ 0loss/FRt3/assignment_loss‚0.9410068446939642 2loss/SRqqt3/assignment_loss‚1.0415769995079707 %loss/total_loss‚39.50209536061226 _step‚760 1loss/FBt3/assignment_loss‚0.17463196535148087 .loss/FRt2/detection_loss‚2.130402731276523 /loss/FRt3/detection_loss‚1.7923457155173474 /loss/FRt4/detection_loss‚2.2835514307022096 /loss/FBt1/detection_loss‚2.2318234627063456 _runtime‚7198.553151496 /loss/FRt1/detection_loss‚3.0869506412538987 epoch‚7 trainer/global_step‚9249 0loss/FBt2/assignment_loss‚0.4514360936232565 2loss/SRqqt2/assignment_loss‚0.8636722262948752 1loss/FBt4/assignment_loss‚0.49581297375594396 /loss/FRt2/assignment_loss‚2.435032943073584 2loss/SRqqt4/assignment_loss‚1.4891204691313682 1loss/SRqqt3/detection_loss‚0.8958417960142685 2loss/SRqqt1/assignment_loss‚1.3735894995076317 1loss/FBt1/assignment_loss‚0.18340890942356333 0loss/SRqqt1/detection_loss‚2.231374795096261 1loss/SRqqt2/detection_loss‚1.8347955100238322 1loss/SRqqt4/detection_loss‚2.0681108962348165 /loss/FBt2/detection_loss‚2.4774498397653755 0loss/FBt3/detection_loss‚0.40888829620719963 % _timestamp‚1.7884120955657785e+09 /loss/FRt1/assignment_loss‚3.176505002482184 0loss/FRt4/assignment_loss‚2.8055119693279265 /loss/FBt4/detection_loss‚2.6292543496404375ø‹8’ž#‚(:‡ ÁäÔ€š§ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xSzŸ#‚(:ï ÁäÔ‹Àê # disk.nvme0c0n1.in‚ 1187.62109375 network.recv‚ 2.603764e+06$ disk.nvme0n1.out‚48773.681640625 disk.rbd1.in‚0$ memory_percent‚88.14055919647217 disk.rbd0.in‚0& disk.nvme0c0n1.out‚48773.681640625! disk.nvme0n1.in‚ 1187.62109375" proc.memory.rssMB‚ 2811.9140625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0! disk.dm-0.out‚48773.681640625 disk.dm-1.out‚ 332.0234375* disk./.usagePercent‚16.598628397650074 disk.sda.out‚ 332.0234375 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 cpu‚25.338820249773637 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 332.0234375 disk.drbd1716.in‚0 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1943.05078125* proc.memory.percent‚17.162561416625977 disk.rbd1.out‚ 33.36328125 network.sent‚ 2.2322764e+07% disk./.usageGB‚247.33155822753906 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0c§8K’ #‚(:‡ ÐäÔã¯Þ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÎã†x¡#‚(:í ÐäÔÈóØ  disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# memory_percent‚88.1375789642334( proc.memory.availableMB‚ 1943.5390625 network.recv‚ 2.609266e+06 disk.rbd1.out‚ 33.3955078125 cpu‚25.33818830345262 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375 disk.dm-0.out‚48865.24609375 disk.dm-1.out‚ 332.87890625 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 332.87890625 disk.drbd2266.in‚0) disk./.usagePercent‚16.60011119328839 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.162561416625977 proc.cpu.threads‚17 network.sent‚ 2.2430768e+07! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375% disk.nvme0c0n1.out‚48865.24609375 disk.drbd1716.in‚0 disk.sda.out‚ 332.87890625 disk.sda1.in‚0# disk.nvme0n1.out‚48865.24609375 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0" proc.memory.rssMB‚ 2811.9140625% disk./.usageGB‚247.35365295410156 disk.rbd1.in‚0 disk.drbd2265.out‚0×OH‚‘¢#‚(:† ßäÔÒùÊ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚143.065' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.613 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô@-r£#‚(:ç ßäÔÿ¯î $ memory_percent‚87.24184036254883 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 333.66015625' proc.memory.availableMB‚ 2090.296875 disk.dm-1.out‚ 333.66015625* disk./.usagePercent‚16.601381758680827% disk./.usageGB‚247.37258529663086 disk.rbd1.out‚ 33.3955078125 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.sda.out‚ 333.66015625 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 network.sent‚ 2.243688e+07 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 network.recv‚ 2.613529e+06! disk.nvme0n1.out‚ 48933.796875 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.162561416625977 cpu‚25.33759124878414 disk.drbd1716.out‚0 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0# disk.nvme0c0n1.out‚ 48933.796875" proc.memory.rssMB‚ 2811.9140625 proc.cpu.threads‚17 disk.dm-0.out‚ 48933.796875 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0ÙŸ>º2¤#- _wandbruntime‚7234  _runtime‚7234÷gYÇ¥#Á & lr-AdamW/pg1‚4.228811508859551e-05 & lr-AdamW/pg2‚4.228811508859551e-05 trainer/global_step‚9299 % _timestamp‚1.7884121316554635e+09 _runtime‚7234.642910881 _step‚761.‰Ö̦#Æ trainer/global_step‚9299 % _timestamp‚1.7884121316554635e+09 _runtime‚7234.642910881 _step‚761 & lr-AdamW/pg1‚4.228811508859551e-05 & lr-AdamW/pg2‚4.228811508859551e-05ù­è§ð2§#- _wandbruntime‚7234  _runtime‚7234н¿<¨#áO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310615704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚533944238069429 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135121842 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44402234 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175473822 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚540342198470144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7784905117696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚526188645530624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚9299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598241658025472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚762 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚533944909546545 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚548126447225344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7756246976561 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135121844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚548126447225344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7784248755200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚548126447225344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7784248755200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚533944238069429 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135121842 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44402234 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175473822 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚540342198470144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7784905117696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚526188645530624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚9299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598241658025472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚762 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚533944909546545 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128úÿ‹œ2ª#- _wandbruntime‚7235  _runtime‚7235Âúˆiñ&«#ëM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884121325858042e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135135860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚526243549661184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123251642 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8166809520640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚9299 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚534000630112277 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135135858 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚548184294070784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598304174320128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175492276 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚540398551508992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123251195 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7756408973769 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310648200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚540398551508992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310628134 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78844532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590137340859392 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚526243566700544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚540398569511936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175512340 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310648200 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135135858 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598303993060864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8166652201472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7785068183040 _step‚763 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚540398569511936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44406663 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7757063411733 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7785724558848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7785724558848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7785068183040 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7785724558848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7785724558848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7785068183040 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884121325858042e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135135860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚526243549661184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123251642 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8166809520640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚9299 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚534000630112277 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135135858 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚548184294070784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598304174320128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175492276 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚540398551508992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123251195 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7756408973769 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310648200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚540398551508992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310628134 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78844532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590137340859392 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚526243566700544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚540398569511936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175512340 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310648200 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135135858 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598303993060864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8166652201472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7785068183040 _step‚763 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚540398569511936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44406663 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064û”$\õ&­#- _wandbruntime‚7235  _ru®‹‘ ntime‚7235^`´^®#Ø /loss/FBt3/detection_loss‚0.6743523675487727 % _timestamp‚1.7884121325874066e+09 _runtime‚7235.574794755 _step‚764 1loss/SRqqt3/detection_loss‚0.8291959282942116 1loss/FBt2/assignment_loss‚0.34922813321463764 1loss/SRqqt2/detection_loss‚1.8141781904158139 /loss/FRt4/detection_loss‚2.4667806342070877 0loss/FRt1/assignment_loss‚2.7796102902468514 /loss/FRt2/assignment_loss‚2.774563961408355 2loss/SRqqt2/assignment_loss‚1.6313205426226236 1loss/FBt3/assignment_loss‚0.14527169383017816 1loss/FBt4/assignment_loss‚0.21798469394426925 /loss/FRt1/detection_loss‚2.8752567435012146 0loss/SRqqt1/detection_loss‚2.530338321129481 1loss/SRqqt4/detection_loss‚1.7025260264770343 %loss/total_loss‚40.66649483684092 epoch‚7 /loss/FRt2/detection_loss‚2.5365195450558535 /loss/FRt4/assignment_loss‚2.521928016529527 2loss/SRqqt1/assignment_loss‚1.1768675329784553 2loss/SRqqt4/assignment_loss‚1.1817504746588663 /loss/FBt4/detection_loss‚2.5578909864028296 trainer/global_step‚9299 0loss/FRt3/assignment_loss‚1.5166657024196215 2loss/SRqqt3/assignment_loss‚0.7767874217471606 0loss/FBt1/assignment_loss‚0.6721225816937577 /loss/FRt3/detection_loss‚1.6742153920765435 /loss/FBt1/detection_loss‚2.8532201184166803 .loss/FBt2/detection_loss‚2.407919538021088qÖÄc¯#Ý 0loss/FRt1/assignment_loss‚2.7796102902468514 /loss/FRt2/assignment_loss‚2.774563961408355 2loss/SRqqt2/assignment_loss‚1.6313205426226236 1loss/FBt3/assignment_loss‚0.14527169383017816 1loss/FBt4/assignment_loss‚0.21798469394426925 /loss/FRt1/detection_loss‚2.8752567435012146 0loss/SRqqt1/detection_loss‚2.530338321129481 1loss/SRqqt4/detection_loss‚1.7025260264770343 %loss/total_loss‚40.66649483684092 epoch‚7 /loss/FRt2/detection_loss‚2.5365195450558535 /loss/FRt4/assignment_loss‚2.521928016529527 2loss/SRqqt1/assignment_loss‚1.1768675329784553 2loss/SRqqt4/assignment_loss‚1.1817504746588663 /loss/FBt4/detection_loss‚2.5578909864028296 trainer/global_step‚9299 0loss/FRt3/assignment_loss‚1.5166657024196215 2loss/SRqqt3/assignment_loss‚0.7767874217471606 0loss/FBt1/assignment_loss‚0.6721225816937577 /loss/FRt3/detection_loss‚1.6742153920765435 /loss/FBt1/detection_loss‚2.8532201184166803 .loss/FBt2/detection_loss‚2.407919538021088 /loss/FBt3/detection_loss‚0.6743523675487727 % _timestamp‚1.7884121325874066e+09 _runtime‚7235.574794755 _step‚764 1loss/SRqqt3/detection_loss‚0.8291959282942116 1loss/FBt2/assignment_loss‚0.34922813321463764 1loss/SRqqt2/detection_loss‚1.8141781904158139 /loss/FRt4/detection_loss‚2.4667806342070877üE#…’°#‚(:‡ îäÔꌫ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.834' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D_”ܱ#‚(:ö îäÔÁ  disk.drbd1716.in‚0 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2811.9140625 network.recv‚ 2.617711e+06 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 33.5517578125 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚49034.04833984375 disk.rbd1.in‚0 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda3.out‚ 334.27734375 disk.drbd2266.in‚0& disk.nvme0n1.out‚49034.04833984375# disk.dm-0.out‚49034.04833984375 disk.dm-1.out‚ 334.27734375 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125$ disk./.usageGB‚247.4006805419922 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375! disk.nvme0n1.in‚ 1187.62109375$ memory_percent‚87.24222183227539* disk./.usagePercent‚16.603267254170508 disk.sda2.out‚0 disk.dm-0.in‚ 1187.62109375 disk.sda.out‚ 334.27734375 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0' proc.memory.availableMB‚ 2090.234375* proc.memory.percent‚17.162561416625977 cpu‚25.33699531638172 network.sent‚ 2.2442666e+07 disk.rbd0.out‚0ÁŠ9ñœ²#‚(:‘ ýäÔÛ¤ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.789999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ØÚ´r³#‚(:ç ýäÔÑߟ  disk.sda.out‚ 334.96875* proc.memory.percent‚17.162561416625977$ disk./.usageGB‚247.4266242980957 disk.dm-0.in‚ 1187.62109375 network.recv‚ 2.624372e+06 disk.drbd2266.out‚0# disk.nvme0c0n1.in‚ 1187.62109375% disk.nvme0c0n1.out‚49108.37890625 disk.drbd1716.out‚0 disk.sda.in‚ 0.1953125 proc.cpu.threads‚17 network.sent‚ 2.2551855e+07 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd2265.in‚0) proc.memory.availableMB‚ 2090.48828125 disk.sda3.out‚ 334.96875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 33.5517578125# disk.nvme0n1.out‚49108.37890625" proc.memory.rssMB‚ 2811.9140625 disk.rbd0.out‚0 disk.drbd1716.in‚0! disk.nvme0n1.in‚ 1187.62109375 cpu‚25.336411411610122 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.drbd2265.out‚0$ memory_percent‚87.24067211151123 disk.sda2.out‚0 disk.dm-1.in‚ 0.1953125* disk./.usagePercent‚16.605008361410874 disk.sda2.in‚0 disk.dm-0.out‚49108.37890625 disk.dm-1.out‚ 334.96875‹¦ô2´#- _wandbruntime‚7271  _runtime‚7271$Q[!ǵ#Á trainer/global_step‚9349 % _timestamp‚1.7884121686864498e+09 _runtime‚7271.673903199 _step‚765 & lr-AdamW/pg1‚4.086581987434659e-05 & lr-AdamW/pg2‚4.086581987434659e-05$­×Ñ̶#Æ & lr-AdamW/pg1‚4.086581987434659e-05 & lr-AdamW/pg2‚4.086581987434659e-05 trainer/global_step‚9349 % _timestamp‚1.7884121686864498e+09 _runtime‚7271.673903199 _step‚765ýÿÑmE2·#- _wandbruntime‚7271  _runtime‚7271­“š”ç'¸#áO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176419912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚550985759339008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚528933852058624 _runtime‚7271.674978369 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8208859613184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚536730951413953 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550985084973568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543159868415488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601366920897024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79244764 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚601367110558208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚536730279936837 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7825890923520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176419912 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135822642 % _timestamp‚1.7884121686871767e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593158227004928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550985084973568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7796427878213 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135822644 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135822642 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7797082315969 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176399874 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44630309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593158250945024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8208693892096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7825234561024 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123874616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550985084973568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543159868415488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601366920897024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79244764 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚601367110558208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚536730279936837 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7825890923520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176419912 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135822642 % _timestamp‚1.7884121686871767e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593158227004928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550985084973568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7796427878213 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135822644 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135822642 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7797082315969 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176399874 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44630309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593158250945024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8208693892096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7825234561024 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123874616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8209718157824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312254986 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7797898751141 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79252934 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601429428095488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528988756189184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7826053988864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79252932 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543216221454336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135836658 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176418328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123887715 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚536786000502361 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528988773228544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275052 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551042931819008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123887265 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7826710364672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7826710364672 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44634781 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275052 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135836658 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8209550352896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593219877742592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¡”ö&¼#ðM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7797898751141 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79252934 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601429428095488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528988756189184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7826053988864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79252932 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543216221454336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135836658 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176418328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123887715 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚536786000502361 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528988773228544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275052 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551042931819008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123887265 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7826710364672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7826710364672 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44634781 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275052 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135836658 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8209550352896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593219877742592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551042931819008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135836660 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176438392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176438392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚7272.364386122 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7826053988864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543216221454336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚767 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8209718157824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312254986 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ÿöøë2½#- _wandbruntime‚7272  _runtime‚7272¶/êƒ]¾#× % _timestamp‚1.7884121693781748e+09 2loss/SRqqt3/assignment_loss‚0.7157702590269384 1loss/SRqqt4/assignment_loss‚1.236682945325111 0loss/SRqqt2/detection_loss‚2.045512589522534 1loss/SRqqt4/detection_loss‚1.6218710568948433 /loss/FBt1/detection_loss‚2.1553761133780847 .loss/FBt3/detection_loss‚0.458271887608219 epoch‚7 /loss/FRt1/assignment_loss‚2.970305322181611 trainer/global_step‚9349 _runtime‚7272.365554107 0loss/FRt3/assignment_loss‚1.3895337310704319 /loss/FRt4/detection_loss‚2.4594578415882298 0loss/SRqqt3/detection_loss‚0.979675881064205 /loss/FBt4/detection_loss‚2.7040521247046336 .loss/FRt1/detection_loss‚2.589423818247659 .loss/FRt2/detection_loss‚2.459873134151418 0loss/FRt4/assignment_loss‚2.5940933503755708 2loss/SRqqt2/assignment_loss‚1.1700356749610767 2loss/FBt3/assignment_loss‚0.061784092339595455 /loss/FRt3/detection_loss‚1.5713732737878507 1loss/SRqqt1/detection_loss‚1.9351416294639174 0loss/FRt2/assignment_loss‚2.3808091866209153 1loss/SRqqt1/assignment_loss‚1.705121001681766 1loss/FBt2/assignment_loss‚0.49359228094625807 &loss/total_loss‚38.153108596579784 1loss/FBt1/assignment_loss‚0.17682691510820475 1loss/FBt4/assignment_loss‚0.12390455435602885 /loss/FBt2/detection_loss‚2.1546199321746826 _step‚768ëØ°Ïb¿#Ü 1loss/FBt1/assignment_loss‚0.17682691510820475 1loss/FBt4/assignment_loss‚0.12390455435602885 /loss/FBt2/detection_loss‚2.1546199321746826 _step‚768 % _timestamp‚1.7884121693781748e+09 2loss/SRqqt3/assignment_loss‚0.7157702590269384 1loss/SRqqt4/assignment_loss‚1.236682945325111 0loss/SRqqt2/detection_loss‚2.045512589522534 1loss/SRqqt4/detection_loss‚1.6218710568948433 /loss/FBt1/detection_loss‚2.1553761133780847 .loss/FBt3/detection_loss‚0.458271887608219 epoch‚7 /loss/FRt1/assignment_loss‚2.970305322181611 trainer/global_step‚9349 _runtime‚7272.365554107 0loss/FRt3/assignment_loss‚1.3895337310704319 /loss/FRt4/detection_loss‚2.4594578415882298 0loss/SRqqt3/detection_loss‚0.979675881064205 /loss/FBt4/detection_loss‚2.7040521247046336 .loss/FRt1/detection_loss‚2.589423818247659 .loss/FRt2/detection_loss‚2.459873134151418 0loss/FRt4/assignment_loss‚2.5940933503755708 2loss/SRqqt2/assignment_loss‚1.1700356749610767 2loss/FBt3/assignment_loss‚0.061784092339595455 /loss/FRt3/detection_loss‚1.5713732737878507 1loss/SRqqt1/detection_loss‚1.9351416294639174 0loss/FRt2/assignment_loss‚2.3808091866209153 1loss/SRqqt1/assignment_loss‚1.705121001681766 1loss/FBt2/assignment_loss‚0.49359228094625807 &loss/total_loss‚38.153108596579784€ §¬ª’À#‚(:‡ Œ‚äÔÄŒ¬ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.136' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8272 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®Mžš‚Á#‚(:÷ Œ‚äÔŒ©­  cpu‚25.33581488334848 network.recv‚ 2.634629e+06 disk.rbd1.out‚ 33.583984375) disk./.usagePercent‚16.60735415844712 disk.sda.in‚ 0.1953125 disk.sda2.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚49184.04931640625 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375 disk.sda3.out‚ 335.68359375 disk.drbd2266.out‚0# disk.dm-0.out‚49184.04931640625 disk.drbd2265.out‚0% disk./.usageGB‚247.46157836914062 disk.sda.out‚ 335.68359375 disk.drbd2080.out‚0 disk.sda1.out‚0$ memory_percent‚86.50281429290771) proc.memory.availableMB‚ 2211.37890625 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚49184.04931640625 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 335.68359375" proc.memory.rssMB‚ 2811.9140625 network.sent‚ 2.2560431e+07 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2265.in‚0* proc.memory.percent‚17.162561416625977 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.drbd1716.in‚0! disk.nvme0n1.in‚ 1187.62109375çÚW’Â#‚(:‡ ›‚äÔÚç gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì½ü€Ã#‚(:õ ›‚äÔ³¢Ñ ! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375) proc.memory.availableMB‚ 2211.07421875' disk.nvme0c0n1.out‚49277.1357421875 disk.rbd1.in‚0" disk.dm-0.out‚49277.1357421875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚49277.1357421875 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 336.12109375* proc.memory.percent‚17.162561416625977 disk.rbd0.out‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 336.12109375" proc.memory.rssMB‚ 2811.9140625 disk.sda2.out‚0 disk.sda.out‚ 336.12109375 disk.drbd2080.out‚0 cpu‚25.335199855266776 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0# memory_percent‚86.5046739578247 network.recv‚ 2.641726e+06 disk.sda2.in‚0 disk.rbd1.out‚ 33.583984375 proc.cpu.threads‚17* disk./.usagePercent‚16.607798843534095% disk./.usageGB‚247.46820449829102# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 disk.sda.in‚ 0.1953125 network.sent‚ 2.2669238e+07û8ñœÄ#‚(:‘ ª‚äÔø†’ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚186.739' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚37.34780000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýC½úÅ#‚(:ö ª‚äÔÅѦ  cpu‚25.334652698128874% disk./.usageGB‚247.52980422973633 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.sda1.in‚0 disk.rbd1.out‚ 33.740234375! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2266.in‚0 disk.drbd2080.out‚0& disk.nvme0n1.out‚49377.40478515625 disk.sda.out‚ 336.7109375( disk.nvme0c0n1.out‚49377.40478515625 proc.cpu.threads‚17 network.recv‚ 2.646767e+06 disk.drbd2080.in‚0$ memory_percent‚86.50157451629639 disk.drbd2266.out‚0* proc.memory.percent‚17.162561416625977 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 336.7109375* disk./.usagePercent‚16.611932853197008 disk.drbd1716.in‚0 disk.drbd1716.out‚0# disk.dm-0.out‚49377.40478515625 disk.sda3.in‚ 0.1953125 disk.rbd1.in‚0 disk.dm-0.in‚ 1187.62109375 disk.sda3.out‚ 336.7109375 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2211.58203125" proc.memory.rssMB‚ 2811.9140625 network.sent‚ 2.2676027e+07 disk.rbd0.out‚0%¬Ä2Æ#- _wandbruntime‚7308  _runtime‚7308Ù±fÉÇ#à _step‚769 ' lr-AdamW/pg1‚3.9463669231343734e-05 ' lr-AdamW/pg2‚3.9463669231343734e-05 trainer/global_step‚9399 % _timestamp‚1.7884122055505393e+09 _runtime‚7308.53799942845ßçÎÈ#È _step‚769 ' lr-AdamW/pg1‚3.9463669231343734e-05 ' lr-AdamW/pg2‚3.9463669231343734e-05 trainer/global_step‚9399 % _timestamp‚1.7884122055505393e+09 _runtime‚7308.537999428ÀI½2É#- _wandbruntime‚7308  _runtime‚7308P"bôç'Ê#áO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8251799779840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚604492742843392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8251955015168 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177325926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539516993281361 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136523444 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚596240787828224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136523442 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177345964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553843722721792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177325926 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177345964 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚7308.538972154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7866220366848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44857194 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136523442 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚553844397087232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7866876729344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545977520357888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79653166 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313849368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884122055512638e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7866876729344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44856756 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚604492563667968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531679075625984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136523444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚604492742843392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8251955015168 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177325926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539516993281361 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136523444 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚596240787828224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136523442 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177345964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553843722721792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177325926 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177345964 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚7308.538972154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7866220366848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44857194 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136523442 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚553844397087232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7866876729344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545977520357888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79653166 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313849368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884122055512638e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7866876729344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44856756 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚604492563667968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531679075625984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136523444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313881838 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136537460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚9399 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚546033873396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177364444 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7867039794688 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124523007 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596302438565888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136537458 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚604555260429312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546033855393792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7867696170496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124522553 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7838734090549 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚546033855393792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚771 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177344380 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539572042369769 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596302414625792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79661332 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8252821863424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136537460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44861673 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531733979756544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7838079652585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚604555068684288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531733962717184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539572713847093 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79661334 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8252654058496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44861221 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _runtime‚7309.220969233 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313901904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546033873396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553901569567232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313881838 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430¸—µÿö&Î#ðM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313881838 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136537460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚9399 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚546033873396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177364444 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7867039794688 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124523007 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596302438565888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136537458 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚604555260429312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546033855393792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7867696170496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124522553 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7838734090549 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚546033855393792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚771 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177344380 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539572042369769 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596302414625792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79661332 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8252821863424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136537460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44861673 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531733979756544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7838079652585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚604555068684288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531733962717184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539572713847093 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79661334 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8252654058496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44861221 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _runtime‚7309.220969233 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313901904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546033873396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553901569567232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313881838 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136537458 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ƒøå:2Ï#- _wandbruntime‚7309  _runtime‚7309e )š[Ð#Õ 0loss/FBt3/assignment_loss‚0.1325103019220456 /loss/FRt1/detection_loss‚2.5945593779151506 1loss/SRqqt1/detection_loss‚2.3483864866398476 0loss/SRqqt2/detection_loss‚1.511226717955791 0loss/FBt4/assignment_loss‚0.1948903573003398 .loss/FBt1/detection_loss‚3.166524112224579 _runtime‚7309.221941007 2loss/SRqqt1/assignment_loss‚1.8141204589927518 2loss/SRqqt3/assignment_loss‚0.8489578217309567 .loss/FRt2/detection_loss‚2.072744075716129 /loss/FRt4/detection_loss‚2.4325213466965874 0loss/FRt1/assignment_loss‚3.6573978855803206 /loss/FRt3/assignment_loss‚1.268735137723741 2loss/SRqqt4/assignment_loss‚1.4614404739439488 1loss/FBt2/assignment_loss‚0.32711868600633065 /loss/FBt3/detection_loss‚0.6036317699929319 /loss/FBt4/detection_loss‚2.7329608542578563 epoch‚7 _step‚772 /loss/FRt3/detection_loss‚1.7561200123635077 .loss/FBt2/detection_loss‚2.348556853257693 /loss/FRt4/assignment_loss‚3.417739495288494 2loss/SRqqt2/assignment_loss‚1.4937972753810196 0loss/SRqqt4/detection_loss‚1.806987874507904 0loss/FRt2/assignment_loss‚2.8765196386310787 1loss/SRqqt3/detection_loss‚1.1538150877292668 %loss/total_loss‚42.03522843398126 trainer/global_step‚9399 2loss/FBt1/assignment_loss‚0.013966332222996211 % _timestamp‚1.7884122062345471e+09ïø?`Ñ#Ú 2loss/FBt1/assignment_loss‚0.013966332222996211 % _timestamp‚1.7884122062345471e+09 0loss/FBt3/assignment_loss‚0.1325103019220456 /loss/FRt1/detection_loss‚2.5945593779151506 1loss/SRqqt1/detection_loss‚2.3483864866398476 0loss/SRqqt2/detection_loss‚1.511226717955791 0loss/FBt4/assignment_loss‚0.1948903573003398 .loss/FBt1/detection_loss‚3.166524112224579 _runtime‚7309.221941007 2loss/SRqqt1/assignment_loss‚1.8141204589927518 2loss/SRqqt3/assignment_loss‚0.8489578217309567 .loss/FRt2/detection_loss‚2.072744075716129 /loss/FRt4/detection_loss‚2.4325213466965874 0loss/FRt1/assignment_loss‚3.6573978855803206 /loss/FRt3/assignment_loss‚1.268735137723741 2loss/SRqqt4/assignment_loss‚1.4614404739439488 1loss/FBt2/assignment_loss‚0.32711868600633065 /loss/FBt3/detection_loss‚0.6036317699929319 /loss/FBt4/detection_loss‚2.7329608542578563 epoch‚7 _step‚772 /loss/FRt3/detection_loss‚1.7561200123635077 .loss/FBt2/detection_loss‚2.348556853257693 /loss/FRt4/assignment_loss‚3.417739495288494 2loss/SRqqt2/assignment_loss‚1.4937972753810196 0loss/SRqqt4/detection_loss‚1.806987874507904 0loss/FRt2/assignment_loss‚2.8765196386310787 1loss/SRqqt3/detection_loss‚1.1538150877292668 %loss/total_loss‚42.03522843398126 trainer/global_step‚9399„†î—Ò#‚(:‡ ¹5â“Úƒ‚äÔÈî– gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žE„îrÓ#‚(:ç ¹‚äÔÉ»¯  disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.out‚ 337.2890625 disk.drbd2266.out‚0' proc.memory.availableMB‚ 2320.796875 proc.cpu.threads‚17 disk.sda3.out‚ 337.2890625 disk.drbd1716.out‚0" disk.nvme0n1.out‚ 49449.3671875 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0$ memory_percent‚85.83498001098633 disk.dm-0.out‚ 49449.3671875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.1953125" proc.memory.rssMB‚ 2811.9140625 network.sent‚ 2.2682275e+07 network.recv‚ 2.650962e+06$ disk.nvme0c0n1.out‚ 49449.3671875 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0 cpu‚25.33403683555099 disk.sda1.in‚0 disk.rbd1.out‚ 33.740234375 disk.drbd2265.out‚0 disk.sda.out‚ 337.2890625* disk./.usagePercent‚16.614350844357904 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2266.in‚0 disk.drbd2080.in‚0* proc.memory.percent‚17.162561416625977% disk./.usageGB‚247.56583404541016Õ »ä’Ô#‚(:‡ È‚äÔ¹û— gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚125.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%Í¢vÕ#‚(:ë È‚äԴΛ $ memory_percent‚85.83517074584961 network.sent‚ 2.2790249e+07 network.recv‚ 2.658867e+06 disk.rbd0.out‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚49536.291015625 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125! disk.nvme0n1.in‚ 1187.62109375! disk.dm-0.out‚49536.291015625 disk.sda3.out‚ 338.296875# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1187.62109375 disk.dm-1.out‚ 338.296875 disk.rbd1.in‚0* proc.memory.percent‚17.162561416625977% disk./.usageGB‚247.59918594360352 disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 33.7724609375" proc.memory.rssMB‚ 2811.9140625 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.drbd1716.in‚0) disk./.usagePercent‚16.61658911822977 disk.drbd2266.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚49536.291015625' proc.memory.availableMB‚ 2320.765625 cpu‚25.333452004318133 disk.sda.out‚ 338.296875 disk.drbd2266.in‚0 disk.sda2.in‚0'Àn22Ö#- _wandbruntime‚7344  _runtime‚7344yQJÉ×#à ' lr-AdamW/pg1‚3.8081955365679406e-05 ' lr-AdamW/pg2‚3.8081955365679406e-05 trainer/global_step‚9449 % _timestamp‚1.7884122418549714e+09 _runtime‚7344.842419877 _step‚773˜$º+ÎØ#È ' lr-AdamW/pg1‚3.8081955365679406e-05 ' lr-AdamW/pg2‚3.8081955365679406e-05 trainer/global_step‚9449 % _timestamp‚1.7884122418549714e+09 _runtime‚7344.842419877 _step‚773…Âlã‰2Ù#- _wandbruntime‚7344  _runtime‚7344]"²4æ'Ú#àO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542303035148769 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7878752994785 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚7344.844035917 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7907862535168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178272016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80061564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178251978 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚548795172300288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315496260 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚548795172300288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137224242 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8294801672192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚534424265114624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125145441 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80061566 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7907206172672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7907206172672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45083877 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556702360470016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚9449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178251978 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788412241856029e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚548795154297344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7907862535168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599323324711424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚556703034835456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚548795154297344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8294969490432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚556703034835456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607618294201856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178272016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45084346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7907206172672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7907206172672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45083877 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556702360470016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚9449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178251978 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788412241856029e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚548795154297344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7907862535168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599323324711424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0†Žþ(D2Ü#- _wandbruntime‚7345  _runtime‚7345еý'±!Ý#ëM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚556759532936704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80069732 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125158506 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45088772 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚548851507336192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599384951508992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8295829614080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607680805063168 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315508690 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556760207315456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178270432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137238258 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚7345.532655198 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125158055 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45088323 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178270432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚775 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7878914991993 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178290496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7908681976320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137238260 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7908681976320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528756 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607680625901056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚548851507336192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542358755714501 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548851525339136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178290496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚548851525339136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599384975449088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884122425443249e+09 H:De0¾y@viceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528756 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315508690 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7879569429957 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚556760207315456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7908025600512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9449 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8295674392064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚534479186284544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÞÕ^áö&Þ#ðM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚548851507336192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599384951508992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8295829614080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607680805063168 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315508690 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556760207315456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178270432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137238258 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚7345.532655198 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125158055 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45088323 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178270432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚775 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7878914991993 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178290496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7908681976320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137238260 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7908681976320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528756 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607680625901056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚548851507336192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542358755714501 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548851525339136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178290496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚548851525339136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599384975449088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884122425443249e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528756 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315508690 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7879569429957 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚556760207315456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7908025600512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9449 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8295674392064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚534479186284544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚556759532936704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80069732 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125158506 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45088772 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2‡ú‰±2ß#- _wandbruntime‚7345  _runtime‚7345xUoBTà#Î 2loss/SRqqt3/assignment_loss‚0.9586795827230582 1loss/FBt1/assignment_loss‚0.20683441461402066 /loss/FBt4/detection_loss‚2.5990694959958396 epoch‚7 _runtime‚ 7345.53392808 0loss/FRt3/assignment_loss‚0.8809224135345883 2loss/SRqqt4/assignment_loss‚1.1717851675789932 .loss/FRt1/detection_loss‚3.055508898364173 .loss/FBt2/detection_loss‚2.831244260072708 0loss/FRt2/assignment_loss‚2.6502476017525853 _step‚776 0loss/FBt4/assignment_loss‚0.2414963603077922 1loss/SRqqt1/detection_loss‚2.3210694466318404 /loss/FBt3/detection_loss‚0.5722140155240208 0loss/FBt2/assignment_loss‚0.3259520645951852 /loss/FRt3/detection_loss‚1.7150394423278394 trainer/global_step‚9449 # _timestamp‚1.78841224254652e+09 2loss/SRqqt2/assignment_loss‚1.1037651602097827 -loss/FRt2/detection_loss‚2.14339515511618 /loss/FRt4/detection_loss‚2.4645415170248164 1loss/SRqqt3/detection_loss‚0.8981016423266668 $loss/total_loss‚39.8703201854956 /loss/FRt1/assignment_loss‚3.138275044935721 2loss/SRqqt1/assignment_loss‚1.6144143416945422 0loss/FBt3/assignment_loss‚0.1971817165619192 1loss/SRqqt2/detection_loss‚1.7336803047877292 0loss/SRqqt4/detection_loss‚1.780987106520554 /loss/FRt4/assignment_loss‚2.916317589061205 .loss/FBt1/detection_loss‚2.349597443233837°D…Yá#Ó 2loss/SRqqt4/assignment_loss‚1.1717851675789932 .loss/FRt1/detection_loss‚3.055508898364173 .loss/FBt2/detection_loss‚2.831244260072708 0loss/FRt2/assignment_loss‚2.6502476017525853 _step‚776 0loss/FBt4/assignment_loss‚0.2414963603077922 1loss/SRqqt1/detection_loss‚2.3210694466318404 /loss/FBt3/detection_loss‚0.5722140155240208 0loss/FBt2/assignment_loss‚0.3259520645951852 /loss/FRt3/detection_loss‚1.7150394423278394 trainer/global_step‚9449 # _timestamp‚1.78841224254652e+09 2loss/SRqqt2/assignment_loss‚1.1037651602097827 -loss/FRt2/detection_loss‚2.14339515511618 /loss/FRt4/detection_loss‚2.4645415170248164 1loss/SRqqt3/detection_loss‚0.8981016423266668 $loss/total_loss‚39.8703201854956 /loss/FRt1/assignment_loss‚3.138275044935721 2loss/SRqqt1/assignment_loss‚1.6144143416945422 0loss/FBt3/assignment_loss‚0.1971817165619192 1loss/SRqqt2/detection_loss‚1.7336803047877292 0loss/SRqqt4/detection_loss‚1.780987106520554 /loss/FRt4/assignment_loss‚2.916317589061205 .loss/FBt1/detection_loss‚2.349597443233837 2loss/SRqqt3/assignment_loss‚0.9586795827230582 1loss/FBt1/assignment_loss‚0.20683441461402066 /loss/FBt4/detection_loss‚2.5990694959958396 epoch‚7 _runtime‚ 7345.53392808 0loss/FRt3/assignment_loss‚0.8809224135345883ˆË]ß’â#‚(:‡ ׂäÔ†Œõ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.151' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N$¯F|ã#‚(:ñ ׂäÔʜà  disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0% disk./.usageGB‚247.63565826416016 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 network.sent‚ 2.2796248e+07 disk.sda.in‚ 0.1953125! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375! disk.dm-0.out‚49604.521484375 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 339.01953125) disk./.usagePercent‚16.61903680626464 disk.sda.out‚ 339.01953125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1187.62109375& disk.nvme0c0n1.out‚49604.521484375 network.recv‚ 2.663109e+06 disk.drbd2266.in‚0' proc.memory.availableMB‚ 2320.984375* proc.memory.percent‚17.162561416625977 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 33.7724609375" proc.memory.rssMB‚ 2811.9140625 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0$ disk.nvme0n1.out‚49604.521484375 disk.drbd2265.in‚0 disk.sda3.out‚ 339.01953125 disk.drbd2080.out‚0 disk.drbd2265.out‚0$ memory_percent‚85.83383560180664 cpu‚25.332880343340204 Gœ8ä#‚(:… æ‚äÔ錦 gpu.0.gpu‚16.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.18' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.636 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è(4ò~å#‚(:ó æ‚äÔî² % disk./.usageGB‚247.66758728027344 disk.sda2.out‚0 disk.dm-1.out‚ 339.3671875 disk.drbd2266.out‚0* proc.memory.percent‚17.162561416625977 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 33.8349609375 disk.sda.in‚ 0.1953125 proc.cpu.threads‚17" disk.dm-0.out‚49669.2197265625 disk.sda.out‚ 339.3671875 network.recv‚ 2.684881e+06 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.in‚0% disk.nvme0n1.out‚49669.2197265625! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.dm-0.in‚ 1187.62109375) proc.memory.availableMB‚ 2320.97265625' disk.nvme0c0n1.out‚49669.2197265625 network.sent‚ 2.2911425e+07 disk.drbd2080.out‚0$ memory_percent‚85.83390712738037" proc.memory.rssMB‚ 2811.9140625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.1953125* disk./.usagePercent‚16.621179589326232 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 339.3671875 cpu‚25.33230828557767­¾f’æ#‚(:‡ õ‚äԈ∠gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.5728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¬˜&ç#‚(:ô õ‚äÔÚ•– ! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 cpu‚25.331763543425332$ disk./.usageGB‚247.6966094970703 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 33.8349609375% disk.nvme0n1.out‚49739.8447265625 disk.sda3.out‚ 339.86328125 disk.drbd2266.out‚0' proc.memory.availableMB‚ 2320.484375 network.sent‚ 2.2918448e+07 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 339.86328125 disk.sda.in‚ 0.1953125 disk.drbd2266.in‚0* disk./.usagePercent‚16.623127294646732 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 339.86328125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚49739.8447265625 disk.drbd2080.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.sda3.in‚ 0.1953125$ memory_percent‚85.83688735961914* proc.memory.percent‚17.162561416625977 disk.sda2.in‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0 network.recv‚ 2.690104e+06 disk.drbd1716.in‚0" disk.dm-0.out‚49739.8447265625" proc.memory.rssMB‚ 2811.9140625FGT´2è#- _wandbruntime‚7381  _runtime‚7381¼cmÉé#à ' lr-AdamW/pg1‚3.6720966224452255e-05 ' lr-AdamW/pg2‚3.6720966224452255e-05 trainer/global_step‚9499 % _timestamp‚1.7884122786475544e+09 _runtime‚7381.634992124 _step‚777+¬QnÎê#È % _timestamp‚1.7884122786475544e+09 _runtime‚7381.634992124 _step‚777 ' lr-AdamW/pg1‚3.6720966224452255e-05 ' lr-AdamW/pg2‚3.6720966224452255e-05 trainer/global_step‚9499‰20Íþ2ë#- _wandbruntime‚7381  _runtime‚7381{Þ‹·ç'ì#áO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317123112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚602405861594624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 % _timestamp‚1.7884122786482625e+09 _step‚778 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚7381.636016016 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137925044 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551612824242688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317103072 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚545089077016177 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137925044 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125780489 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610743978150400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80469966 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7948191978496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80469964 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559561672583680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179178030 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179198068 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610743788489216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚551612824242688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545088405539061 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137925042 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7948848340992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7948191978496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137925042 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559560998218240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602405837654528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179178030 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125780941 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7919588334193 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537169471642624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559560998218240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8338116555776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7918933896437 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45310975 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8337950834688 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179198068 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45310525 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551612806239744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚545089077016177 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137925044 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125780489 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610743978150400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80469966 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7948191978496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80469964 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559561672583680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179178030 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179198068 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610743788489216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_baÕø#‡tch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚551612824242688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545088405539061 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137925042 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7948848340992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7948191978496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137925042 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559560998218240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602405837654528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179178030 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125780941 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7919588334193 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537169471642624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559560998218240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8338116555776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7918933896437 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45310975 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8337950834688 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179198068 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45310525 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551612806239744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179196484 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7949011406336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545144126104585 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610806488623616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80478134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559618170684928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45315038 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137939060 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551669177281536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551669159278592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7949667782144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317135542 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602467512332288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8338976291328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45315476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179216548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7949667782144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80478132 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125793170 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545144797581909 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179196484 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537224392812544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884122793353083e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610806305267200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137939058 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚9499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317135542 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125793610 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚551669177281536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚537224375773184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8338816875008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064ਭö&ð#ðM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179196484 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7949011406336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545144126104585 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610806488623616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80478134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559618170684928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45315038 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137939060 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551669177281536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551669159278592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7949667782144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317135542 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602467512332288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8338976291328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45315476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179216548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7949667782144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80478132 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125793170 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545144797581909 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179196484 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537224392812544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884122793353083e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610806305267200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137939058 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚9499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317135542 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125793610 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚551669177281536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚537224375773184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8338816875008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430‹FÅô42ñ#- _wandbruntime‚7382  _runtime‚7382>vçŸZò#Ô _runtime‚7382.324412303 _step‚780 0loss/FRt3/assignment_loss‚1.0267587105433147 1loss/FBt2/assignment_loss‚0.35936987237073365 0loss/FBt3/assignment_loss‚0.1585642472012858 1loss/SRqqt1/detection_loss‚2.3019124964872995 0loss/SRqqt3/detection_loss‚1.124916409713126 epoch‚7 trainer/global_step‚9499 /loss/FRt1/assignment_loss‚3.216781587600708 -loss/FBt2/detection_loss‚3.26759002606074 2loss/SRqqt2/assignment_loss‚1.7822861107752512 /loss/FRt1/detection_loss‚3.1403429198265074 .loss/FRt3/detection_loss‚1.494276655320492 /loss/FBt1/detection_loss‚2.6860346233143524 /loss/FBt4/detection_loss‚2.1608847975730896 0loss/FRt2/assignment_loss‚3.3174720989032225 /loss/FRt4/detection_loss‚2.2360522116844854 % _timestamp‚1.7884122793370225e+09 2loss/SRqqt3/assignment_loss‚0.9475139993873816 1loss/SRqqt2/detection_loss‚2.1487828448022666 %loss/total_loss‚42.27309829957764 2loss/SRqqt1/assignment_loss‚1.6702963581515684 2loss/SRqqt4/assignment_loss‚1.5352662497920528 1loss/FBt1/assignment_loss‚0.16791887171220435 1loss/FBt4/assignment_loss‚0.08347814045919222 1loss/SRqqt4/detection_loss‚1.7130075530061182 /loss/FRt4/assignment_loss‚2.934392330547174 .loss/FRt2/detection_loss‚2.329199576242513 .loss/FBt3/detection_loss‚0.469999608102565f_Œ_ó#Ù 0loss/FRt2/assignment_loss‚3.3174720989032225 /loss/FRt4/detection_loss‚2.2360522116844854 % _timestamp‚1.7884122793370225e+09 2loss/SRqqt3/assignment_loss‚0.9475139993873816 1loss/SRqqt2/detection_loss‚2.1487828448022666 %loss/total_loss‚42.27309829957764 2loss/SRqqt1/assignment_loss‚1.6702963581515684 2loss/SRqqt4/assignment_loss‚1.5352662497920528 1loss/FBt1/assignment_loss‚0.16791887171220435 1loss/FBt4/assignment_loss‚0.08347814045919222 1loss/SRqqt4/detection_loss‚1.7130075530061182 /loss/FRt4/assignment_loss‚2.934392330547174 .loss/FRt2/detection_loss‚2.329199576242513 .loss/FBt3/detection_loss‚0.469999608102565 _runtime‚7382.324412303 _step‚780 0loss/FRt3/assignment_loss‚1.0267587105433147 1loss/FBt2/assignment_loss‚0.35936987237073365 0loss/FBt3/assignment_loss‚0.1585642472012858 1loss/SRqqt1/detection_loss‚2.3019124964872995 0loss/SRqqt3/detection_loss‚1.124916409713126 epoch‚7 trainer/global_step‚9499 /loss/FRt1/assignment_loss‚3.216781587600708 -loss/FBt2/detection_loss‚3.26759002606074 2loss/SRqqt2/assignment_loss‚1.7822861107752512 /loss/FRt1/detection_loss‚3.1403429198265074 .loss/FRt3/detection_loss‚1.494276655320492 /loss/FBt1/detection_loss‚2.6860346233143524 /loss/FBt4/detection_loss‚2.1608847975730896Œ•·Ül’ô#‚(:‡ „ƒäÔü¦ð gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.152' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å•Ñõ#‚(:ö „ƒäÔáùÖ * proc.memory.percent‚17.162561416625977 disk.dm-1.out‚ 340.6171875 disk.rbd1.out‚ 33.8505859375 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 340.6171875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚49834.25732421875 disk.drbd2265.in‚0 disk.sda.out‚ 340.6171875 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2811.9140625 network.sent‚ 2.2924058e+07$ disk./.usageGB‚247.7192840576172 disk.sda3.in‚ 0.1953125 disk.drbd2080.out‚0$ memory_percent‚85.22975444793701* disk./.usagePercent‚16.624649003430257& disk.nvme0n1.out‚49834.25732421875 disk.dm-0.in‚ 1187.62109375 cpu‚25.331163765764874 proc.cpu.threads‚17 network.recv‚ 2.693693e+06! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.out‚0 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚49834.25732421875 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0) proc.memory.availableMB‚ 2419.95703125¶äçµ’ö#‚(:‡ “ƒäÔ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.851' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Ð~ƒ÷#‚(:ì “ƒäÔ‚Ù’ * proc.memory.percent‚17.162561416625977 disk.drbd1716.out‚0) proc.memory.availableMB‚ 2419.95703125 disk.dm-0.in‚ 1187.62109375* disk./.usagePercent‚16.626152535678848 disk.drbd2080.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚341.1875 disk.drbd2265.out‚0 network.sent‚ 2.3032721e+07 network.recv‚ 2.701804e+06 disk.drbd2265.in‚0 disk.sda.out‚341.1875 disk.rbd0.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2811.9140625 cpu‚25.33060021602769! disk.nvme0n1.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda2.in‚0( disk.nvme0c0n1.out‚49913.74951171875 disk.rbd1.in‚0 disk.rbd1.out‚ 33.8505859375 proc.cpu.threads‚17$ disk./.usageGB‚247.7416877746ãƒì ô582 disk.sda3.out‚341.1875 disk.drbd2266.out‚0# disk.dm-0.out‚49913.74951171875 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚49913.74951171875$ memory_percent‚85.22975444793701×÷¦2ø#- _wandbruntime‚7418  _runtime‚7418ApýÇù#Á & lr-AdamW/pg2‚3.538098543575945e-05 trainer/global_step‚9549 % _timestamp‚1.7884123154708397e+09 _runtime‚7418.458245668 _step‚781 & lr-AdamW/pg1‚3.538098543575945e-05½§#´Ìú#Æ & lr-AdamW/pg2‚3.538098543575945e-05 trainer/global_step‚9549 % _timestamp‚1.7884123154708397e+09 _runtime‚7418.458245668 _step‚781 & lr-AdamW/pg1‚3.538098543575945e-05{ nì2û#- _wandbruntime‚7418  _runtime‚7418¡5!æç'ü#áO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381087705088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80878366 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554430476185088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚613869462242816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554430458182144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7989834146816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45538835 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138625844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884123154715347e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8381253426176 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605488398477824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562419635966464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚554430476185088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554430458182144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562420310331904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180104082 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318729924 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7960423673601 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180124120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7989834146816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613869651904000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547874447406469 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562419635966464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7418.459205569 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7989177784320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚9549 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚605488374537728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547875118883585 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚539914695209984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562420310331904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7989177784320 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138625844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _step‚782 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138625842 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126417652 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180104082 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7959769235845 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45539286Ë?9 ì'ý#æO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7989834146816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45538835 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138625844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884123154715347e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8381253426176 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605488398477824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562419635966464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚554430476185088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554430458182144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562420310331904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180104082 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318729924 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7960423673601 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180124120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7989834146816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613869651904000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547874447406469 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562419635966464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7418.459205569 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7989177784320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚9549 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚605488374537728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547875118883585 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚539914695209984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562420310331904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7989177784320 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138625844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _step‚782 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138625842 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126417652 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180104082 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7959769235845 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45539286 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381087705088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80878366 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554430476185088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚613869462242816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554430458182144Ž72¥!2þ#- _wandbruntime‚7419  _runtime‚7419Ö:šð&ÿ#êM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762394 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180142600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126429850 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788412316152748e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562477482811904 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613932166222336 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562477482811904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782460 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8382117006848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562476808433152 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138639858 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚783 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539969582301184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7960585670809 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554486829223936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605550049215488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7961240108773 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318762394 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613931984963072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554486811220992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138639860 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7419.140612184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45543762 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547930167971993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605550025275392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138639858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45543318 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088B'¾_Ú€$ïM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126430296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80886534 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554486829223936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180122536 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126429850 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788412316152748e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562477482811904 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613932166222336 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562477482811904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782460 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8382117006848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562476808433152 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138639858 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚783 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539969582301184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7960585670809 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554486829223936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605550049215488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7961240108773 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318762394 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613931984963072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554486811220992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138639860 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7419.140612184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45543762 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547930167971993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605550025275392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138639858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45543318 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762394 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180142600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚565278273714688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45765826 _step‚786 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616995279953408 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542659884698624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8001259013009 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542659901737984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181050172 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127052590 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550660489273877 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139326644 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45766271 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚9599 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181050172 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚557248128127488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81286766 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚557248110124544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚550661160750993 _runtime‚7455.445927196 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181030134 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚565278948080128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320376816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616995098680832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608570911420928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8000604575253 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8424187259904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139326644 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81286764 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8030163590144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚565278273714688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557248110124544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8030163590144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557248128127488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚550661160750993 _runtime‚7455.445927196 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181030134 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚565278948080128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320376816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616995098680832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608570911420928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8000604575253 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8424187259904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139326644 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81286764 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8030163590144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚565278273714688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557248110124544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8030163590144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557248128127488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚565278273714688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45765826 _step‚786 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616995279953408 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542659884698624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8001259013009 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542659901737984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181050172 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127052590 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550660489273877 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139326644 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45766271 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚9599 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181050172 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚557248128127488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81286766 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944’V^²Ú2$- _wandbruntime‚7456  _runtime‚7456:òéñ&‘$ëM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181068652 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127065804 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139340660 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557304481166336 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127065343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45770411 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8030983017984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542714805868544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8001421010217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚565335446181376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884123531492774e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181048588 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8002075448181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617057797740032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8031639393792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320389246 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚617057612286464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45770870 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139340660 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557304481166336 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127065343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45770411 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8030983017984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542714805868544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8001421010217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚565335446181376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884123531492774e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181048588 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8002075448181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617057797740032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8031639393792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320389246 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚617057612286464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45770870 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620120883340288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚560065780069888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8467600757248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚9649 _runtime‚7492.536183344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚620121073001472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8042094352417 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMo»ˆ1¶Ånitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45994177 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127688882 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181956186 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568137585828352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181956186 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚611653448304128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81695164 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568137585828352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127689343 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545405108265984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚560065762066944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568136911462912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553446531141285 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8071805758464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚560065780069888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8071805758464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8071149395968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181976224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140027442 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45993718 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81695166 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140027442 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568136911462912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545405091226624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181976224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140027444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚611653472244224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚790 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8041439914661 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884123895485003e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8071149395968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚553447202618401 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚560065762066944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620120883340288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚560065780069888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8467600757248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚9649 _runtime‚7492.536183344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚620121073001472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8042094352417 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45994177 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127688882 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181956186 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568137585828352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181956186 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚611653448304128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81695164 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568137585828352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127689343 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545405108265984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚560065762066944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568136911462912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553446531141285 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8071805758464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚560065780069888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8071805758464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8071149395968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181976224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140027442 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45993718 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81695166 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140027442 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568136911462912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545405091226624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181976224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140027444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚611653472244224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚790 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8041439914661 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884123895485003e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8071149395968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚553447202618401 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚560065762066944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚620183585304576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553502923184133 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788412390235512e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036164 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚560122115105792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8042910787589 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8072625199616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016098 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181994704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8468462322688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620183401948160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚611715099041792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚553502251706809 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚560122133108736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071968823808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181974640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127701983 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚545460012396544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81703332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568194083929600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140041458 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322036164 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181994704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81703334 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127701529 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611715122981888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8468302906368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8042256349625 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181974640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚560122115105792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140041458 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322016098 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45998649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140041460 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844²Ë} ¢$ïM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016098 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181994704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8468462322688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620183401948160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚611715099041792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚553502251706809 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚560122133108736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071968823808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181974640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127701983 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚545460012396544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81703332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568194083929600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140041458 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322036164 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181994704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81703334 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127701529 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611715122981888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8468302906368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8042256349625 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181974640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚560122115105792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140041458 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/request/”éGëed_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322016098 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45998649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140041460 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚791 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚560122133108736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚545459995357184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚620183585304576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553502923184133 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788412390235512e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036164 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚560122115105792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8042910787589 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8072625199616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9649—µ-Ä"2£$- _wandbruntime‚7493  _runtime‚7493 -oY¤$Ó /loss/FRt1/assignment_loss‚2.718237013920494 2loss/SRqqt2/assignment_loss‚1.4348160250363156 2loss/SRqqt3/assignment_loss‚0.9604620112679966 1loss/FBt3/assignment_loss‚0.09697499135331439 -loss/FRt2/detection_loss‚2.10812618494368 /loss/FRt4/detection_loss‚2.0821537197782445 1loss/SRqqt1/detection_loss‚2.7265534661710262 /loss/FRt4/assignment_loss‚2.636312724306033 0loss/FBt4/assignment_loss‚0.0913768270434553 /loss/FBt2/detection_loss‚2.7829377386305065 /loss/FBt3/detection_loss‚0.5983014075020545 % _timestamp‚1.7884123902372746e+09 0loss/SRqqt4/detection_loss‚1.556845360858874 0loss/FRt3/assignment_loss‚1.2782123764355977 2loss/FBt2/assignment_loss‚0.023564262470851343 0loss/SRqqt3/detection_loss‚1.081393472672928 1loss/SRqqt1/assignment_loss‚1.640747596975416 trainer/global_step‚9649 %loss/total_loss‚40.62350532530176 epoch‚7 0loss/FBt1/assignment_loss‚0.4441639710275922 .loss/FRt3/detection_loss‚1.534961299908658 /loss/FBt1/detection_loss‚2.7221312895417213 /loss/FBt4/detection_loss‚2.4064665105607776 1loss/SRqqt2/detection_loss‚1.5513713411413705 _step‚792 0loss/FRt2/assignment_loss‚3.6179698435765393 1loss/SRqqt4/assignment_loss‚1.164814228225838 /loss/FRt1/detection_loss‚3.3646116619524746 _runtime‚7493.224663021;]¨¬^¥$Ø 0loss/FRt2/assignment_loss‚3.6179698435765393 1loss/SRqqt4/assignment_loss‚1.164814228225838 /loss/FRt1/detection_loss‚3.3646116619524746 _runtime‚7493.224663021 /loss/FRt1/assignment_loss‚2.718237013920494 2loss/SRqqt2/assignment_loss‚1.4348160250363156 2loss/SRqqt3/assignment_loss‚0.9604620112679966 1loss/FBt3/assignment_loss‚0.09697499135331439 -loss/FRt2/detection_loss‚2.10812618494368 /loss/FRt4/detection_loss‚2.0821537197782445 1loss/SRqqt1/detection_loss‚2.7265534661710262 /loss/FRt4/assignment_loss‚2.636312724306033 0loss/FBt4/assignment_loss‚0.0913768270434553 /loss/FBt2/detection_loss‚2.7829377386305065 /loss/FBt3/detection_loss‚0.5983014075020545 % _timestamp‚1.7884123902372746e+09 0loss/SRqqt4/detection_loss‚1.556845360858874 0loss/FRt3/assignment_loss‚1.2782123764355977 2loss/FBt2/assignment_loss‚0.023564262470851343 0loss/SRqqt3/detection_loss‚1.081393472672928 1loss/SRqqt1/assignment_loss‚1.640747596975416 trainer/global_step‚9649 %loss/total_loss‚40.62350532530176 epoch‚7 0loss/FBt1/assignment_loss‚0.4441639710275922 .loss/FRt3/detection_loss‚1.534961299908658 /loss/FBt1/detection_loss‚2.7221312895417213 /loss/FBt4/detection_loss‚2.4064665105607776 1loss/SRqqt2/detection_loss‚1.5513713411413705 _step‚792˜Þ–›¦$‚(:’ íƒäÔŠì« gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.051' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.410199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_pIçm§$‚(:â íƒäÔ¾ÿ¼  disk.rbd1.out‚34.125 disk.dm-0.out‚ 50454.2578125 disk.sda.out‚ 345.7734375 disk.sda3.out‚ 345.7734375 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 network.recv‚ 2.766602e+06 disk.drbd1716.out‚0 disk.dm-1.out‚ 345.7734375 disk.drbd2080.in‚0 network.sent‚ 2.3290332e+07 disk.dm-0.in‚ 1187.62109375 disk.sda1.in‚0$ memory_percent‚84.40718650817871" proc.memory.rssMB‚ 2811.9140625 cpu‚25.32720866705305" disk.nvme0n1.out‚ 50454.2578125 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125% disk./.usageGB‚246.57327270507812 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375# disk.nvme0c0n1.in‚ 1187.62109375$ disk.nvme0c0n1.out‚ 50454.2578125 proc.cpu.threads‚17* disk./.usagePercent‚16.547739219993787 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.162561416625977 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 2554.7265625Ü4ﺒ¨$‚(:‡ üƒäÔ˜÷» gpu.0.gpu‚55.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 E~t©$‚(:é üƒäÔÄ¿²  disk.sda1.in‚0 cpu‚25.330463795667498 proc.cpu.threads‚17 disk.dm-1.out‚ 346.5234375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 2554.71875" disk.dm-0.out‚50527.0068359375 disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.162561416625977 network.recv‚ 2.77288e+06% disk./.usageGB‚246.58905029296875 disk.drbd2080.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚50527.0068359375 disk.rbd1.out‚34.125 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd1716.in‚0 network.sent‚ 2.3398491e+07 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.sda.out‚ 346.5234375# disk.nvme0c0n1.in‚ 1187.62109375! disk.nvme0n1.in‚ 1187.62109375$ memory_percent‚84.40723419189453* disk./.usagePercent‚16.548798067155403 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125" proc.memory.rssMB‚ 2811.9140625 disk.sda3.out‚ 346.5234375 disk.drbd1716.out‚0% disk.nvme0n1.out‚50527.0068359375nyöª$‚(:„ ‹„äÔ›¾¹ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.816 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âzy.ƒ«$‚(:ø ‹„äÔ§ƒ£  disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 34.1572265625* proc.memory.percent‚17.162561416625977 network.sent‚ 2.3404561e+07 network.recv‚ 2.777062e+06 disk.sda.out‚ 347.17578125 disk.sda2.out‚0( disk.nvme0c0n1.out‚50596.72998046875! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0$ memory_percent‚84.40573215484619 disk.sda3.in‚ 0.1953125 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 347.17578125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0) disk./.usagePercent‚16.55080286215604% disk./.usageGB‚246.61892318725586 disk.drbd2266.out‚0& disk.nvme0n1.out‚50596.72998046875# disk.dm-0.out‚50596.72998046875) proc.memory.availableMB‚ 2554.96484375 cpu‚25.33950407713843 proc.cpu.threads‚17 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 347.17578125 disk.rbd0.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2811.9140625¥Òû¾‘¬$‚(:† š„äÔÄÒá gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.978' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.5956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`j j}­$‚(:ò š„äÔ‰ðò  disk.rbd1.in‚0 disk.rbd1.out‚ 34.1572265625 disk.drbd2265.out‚0 disk.dm-1.out‚ 348.15234375 disk.sda3.out‚ 348.15234375 disk.drbd2266.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 1187.62109375 network.sent‚ 2.3410388e+07 disk.drbd2080.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375) proc.memory.availableMB‚ 2684.33203125 proc.cpu.threads‚17 network.recv‚ 2.781447e+06 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.drbd1716.in‚0* disk./.usagePercent‚16.552242904536143# disk./.usageGB‚246.640380859375 disk.sda2.in‚0 cpu‚25.348741503163087$ disk.nvme0n1.out‚50688.306640625! disk.dm-0.out‚50688.306640625 disk.rbd0.out‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚50688.306640625 disk.drbd1716.out‚0 disk.drbd2265.in‚0$ memory_percent‚83.61613750457764 disk.drbd2266.out‚0 disk.sda.in‚ 0.1953125 disk.sda.out‚ 348.15234375" proc.memory.rssMB‚ 2811.9140625* proc.memory.percent‚17.162561416625977 disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0 disk.sda1.out‚0Aº\˜2®$- _wandbruntime‚7557  _runtime‚7557M*¯$ÇT jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚614927350152192 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚82201724 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 182758602 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚571275133534720 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 140879306 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 182758602 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8078708761657 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8078217847265 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚563166553371648 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8108069329920 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚82201726 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 $ _timestamp‚1.788412454732124e+09 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚46318161 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 trainer/global_step‚9671 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚563166553371648 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 _step‚793 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚623432825188352 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚623432685904384 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚556475498291681 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 128519733 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚571274622701568 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8108562160128 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚571274622701568 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 8505475036160 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 323621929 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 140879304 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚548397297483776 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323637908 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚571275133534720 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 140879304 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _runtime‚7557.720055561 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚556476006245433 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 140879306 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8108069329920 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚614927326212096 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚563166571374592 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 128519887 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115343872 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 182742625 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚563166571374592 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚46318009 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8505359692288 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 323637908 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚154 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889ÿùG°$ÌT RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 323621929 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚548397280444416 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 323621929 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 140879304 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚548397297483776 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323637908 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚571275133534720 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 140879304 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _runtime‚7557.720055561 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚556476006245433 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 140879306 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8108069329920 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚614927326212096 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚563166571374592 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 128519887 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115343872 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 182742625 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚563166571374592 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚46318009 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8505359692288 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 323637908 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚154 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚614927350152192 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚82201724 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 182758602 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚571275133534720 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 140879306 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 182758602 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_a kÀ4>lloc_retries‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8078708761657 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8078217847265 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚563166553371648 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8108069329920 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚82201726 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 $ _timestamp‚1.788412454732124e+09 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚46318161 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 trainer/global_step‚9671 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚563166553371648 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 _step‚793 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚623432825188352 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚623432685904384 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚556475498291681 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 128519733 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚571274622701568 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8108562160128 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚571274622701568 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 8505475036160 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 128530751 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚614965225885696 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 _runtime‚7558.642023393 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8108434456576 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _step‚794 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 182765904 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚563203383400448 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9671 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8108927286784 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8505724818944 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8078581456673 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚571311817857024 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚614965201945600 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414¹ÚÆÜ])³$×R LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8078581456673 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚571311817857024 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚614965201945600 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚82208228 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚571311817857024 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 182749927 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚46322523 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚548432591243264 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8108434456576 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884124556541905e+09 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182765904 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 323640037 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚556511663614329 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚563203401403392 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚82208230 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚46322676 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚556511155660577 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 128530906 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚155 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 323656016 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 128530751 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚614965225885696 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 _runtime‚7558.642023393 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8108434456576 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _step‚794 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 182765904 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚563203383400448 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9671 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8108927286784 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8505724818944šdœ:´$‚(:’ ©„äÔ‹ß³ gpu.0.gpu‚55.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.737' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.547399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óõ³Xmµ$‚(:â ©„äÔÖ«Ð  network.sent‚ 2.3416739e+07 disk.dm-1.out‚348.75 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚50779.767578125 disk.rbd1.in‚0! disk.dm-0.out‚50779.767578125 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚348.75 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚16.553515517988853! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.drbd2265.out‚0% disk./.usageGB‚246.65934371948242 disk.rbd0.out‚0* proc.memory.percent‚17.162561416625977 disk.rbd1.out‚34.22021484375 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda.out‚348.75 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0$ memory_percent‚83.23547840118408 cpu‚25.35771078649547 disk.sda2.out‚0$ disk.nvme0n1.out‚50779.767578125 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2811.9140625 network.recv‚ 2.786084e+06# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0) proc.memory.availableMB‚ 2746.69921875=P¶2¶$- _wandbruntime‚7571  _runtime‚7571LÄjog·$¥§ =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _step‚795 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16628967225551605 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3277834355831146 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166567325592 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 jet/accuracy_5_of_7‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.02131783217191696 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 *jet/accuracy_1_of_2‚0.7370837330818176 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3543851673603058 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5088329911231995 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17868895828723907 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19797030091285706 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4344947636127472 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37586650252342224 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.580926239490509 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16592943668365479 particle/accuracy_4_of_5‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18009687960147858 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29526567459106445 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 )jet/accuracy_2_of_4‚0.369784951210022 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15678198635578156 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4782078266143799 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 (particle/accuracy‚0.8538618683815002 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24916110932826996 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1884889453649521 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9905454516410828 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48355263471603394 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5633686780929565 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.516905665397644 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4583333134651184 particle/accuracy_3_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9924482107162476 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_2_of_2‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4630506932735443 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16563262045383453 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.42902523279190063 ("Purity/3FRt2Síbu¶E{Rqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.196675643324852 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9460893273353577 -jet/accuracy_5_of_5‚0.0026881720405071974 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5101527571678162 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3913942873477936 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014153439551591873 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23989899456501007 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6510269641876221 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3055555522441864 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 particle/accuracy_5_of_6‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4504068195819855 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 jet/accuracy_4_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46616673469543457 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.013207548297941685 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10768849402666092 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3516278862953186 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16261771321296692 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5097315907478333 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42318424582481384 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 particle/accuracy_4_of_4‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5271851420402527 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9965744018554688 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9915668964385986 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26559048891067505 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 particle/accuracy_1_of_2‚1 jet/accuracy_6_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1571839153766632 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4032737910747528 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.388552188873291 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11149377375841141 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5443349480628967 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +particle/specificity‚0.9507817625999451 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 *jet/accuracy_2_of_6‚0.4551282227039337 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4907848536968231 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4502054750919342 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_6‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5486898422241211 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29976755380630493 ,jet/accuracy_4_of_5‚0.025179212912917137 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5801843404769897 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3261905014514923 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5279174447059631 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46888843178749084 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6225729584693909 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 _runtime‚7571.151433863 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42088350653648376 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5555084943771362 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17500722408294678 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04054022952914238 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5017039775848389 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5009503364562988 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4495305120944977 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08773097395896912 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.445242315530777 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.39726522564888 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024872079491615295 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005885560996830463 ( Purity/2FRt4SRqqt0FBt/FRt_purity‚0.5 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24727505445480347 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8131499290466309 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23468345403671265 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18924285471439362 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.47531822323799133 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919971227645874 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15589384734630585 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.38398686051368713 jet/accuracy_2_of_7‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02825211174786091 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19551584124565125 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4582873582839966 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18957103788852692 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2805555760860443 particle/accuracy_2_of_6‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.47518524527549744 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.029051460325717926 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47053858637809753 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15016891062259674 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2410714328289032 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_5‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5375174880027771 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4797461926937103 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9901708960533142 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983689785003662 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ( Purity/2FRt4SRqqt*FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.474642813205719 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02211165614426136 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5028092265129089 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 jet/accuracy_3_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40086838603019714 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46888843178749084 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.997774064540863 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45603445172309875 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4704861640930176 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23055203258991241 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2465517222881317 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3759258985519409 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_2_of_3‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40588653087615967 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26919281482696533 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41296303272247314 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 +jet/accuracy_2_of_3‚0.23717811703681946 particle/accuracy_6_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9231677651405334 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3684028089046478 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19870202243328094 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.576785683631897 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5264991521835327 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864625334739685 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.42635664343833923 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 particle/accuracy_7_of_7‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.42279699444770813 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.0416666679084301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1608186960220337 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5044039487838745 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 jet/accuracy_1_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5702381730079651 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895066618919373 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12643678486347198 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5115740895271301 ,jet/accuracy_4_of_6‚0.057692307978868484 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2826535105705261 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994331002235413 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3055555522441864 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2545045018196106 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 +particle/sensitivity‚0.3301701843738556 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894058704376221 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_3_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9923855066299438 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28301432728767395 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014202487654983997 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16537542641162872 *jet/accuracy_3_of_4‚0.0766497328877449 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4597330391407013 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9825748205184937 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3581870198249817 particle/accuracy_1_of_4‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 *jet/accuracy_1_of_4‚0.9193448424339294 +jet/accuracy_3_of_6‚0.23076923191547394 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16262322664260864 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.45156538486480713 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20620954036712646 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197531044483185 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19800809025764465 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9989169239997864 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.034088581800460815 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06583906710147858 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25922921299934387 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 trainer/global_step‚9671 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01613685116171837 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1597980260848999 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1309524029493332 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 particle/accuracy_4_of_6‚1 )loss/val_total_loss‚41.36671654559099 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1690274327993393 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04028453677892685 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982808530330658 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3919958472251892 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05714879184961319 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 epoch‚7 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4578881561756134 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5836805701255798 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7884124681616676e+09 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_2_of_4‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9952948093414307 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05818965658545494 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28545767068862915 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2986111044883728 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48256006836891174 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 'particle/f_score‚0.4135001599788666 particle/accuracy_4_of_7‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19213373959064484 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.003250189358368516 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5246913433074951 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5783334374427795 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6992844343185425 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44550055265426636 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987711489200592 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28250229358673096 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.856837272644043 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1205182746052742 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2306390106678009 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5348114371299744 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47139081358909607 +jet/accuracy_3_of_5‚0.10277777165174484 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.522988498210907 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.25154319405555725 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19213373959064484 ,jet/accuracy_4_of_4‚0.009559795260429382 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5752089023590088 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10684524476528168 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11195956915616989 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21792539954185486 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444465637207 jet/accuracy_6_of_6‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1719229519367218 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3908408284187317 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 particle/accuracy_1_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21044299006462097 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3780864179134369 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03938649594783783 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42289644479751587 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017756935209035873 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903900623321533 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742427945137024 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.585023045539856 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9915685653686523 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17965859174728394 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21881742775440216à`œ–­¸$ª§ ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_3_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9923855066299438 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28301432728767395 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014202487654983997 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16537542641162872 *jet/accuracy_3_of_4‚0.0766497328877449 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4597330391407013 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9825748205184937 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3581870198249817 particle/accuracy_1_of_4‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 *jet/accuracy_1_of_4‚0.9193448424339294 +jet/accuracy_3_of_6‚0.230769231915 ÖΆù47394 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16262322664260864 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.45156538486480713 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20620954036712646 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197531044483185 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19800809025764465 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9989169239997864 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.034088581800460815 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06583906710147858 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25922921299934387 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 trainer/global_step‚9671 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01613685116171837 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1597980260848999 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1309524029493332 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 particle/accuracy_4_of_6‚1 )loss/val_total_loss‚41.36671654559099 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1690274327993393 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04028453677892685 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982808530330658 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3919958472251892 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05714879184961319 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 epoch‚7 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4578881561756134 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5836805701255798 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7884124681616676e+09 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_2_of_4‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9952948093414307 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05818965658545494 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28545767068862915 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2986111044883728 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48256006836891174 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 'particle/f_score‚0.4135001599788666 particle/accuracy_4_of_7‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19213373959064484 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.003250189358368516 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5246913433074951 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5783334374427795 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6992844343185425 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44550055265426636 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987711489200592 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28250229358673096 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.856837272644043 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1205182746052742 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2306390106678009 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5348114371299744 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47139081358909607 +jet/accuracy_3_of_5‚0.10277777165174484 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.522988498210907 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.25154319405555725 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19213373959064484 ,jet/accuracy_4_of_4‚0.009559795260429382 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5752089023590088 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10684524476528168 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11195956915616989 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21792539954185486 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444465637207 jet/accuracy_6_of_6‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1719229519367218 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3908408284187317 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 particle/accuracy_1_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21044299006462097 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3780864179134369 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03938649594783783 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42289644479751587 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017756935209035873 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903900623321533 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742427945137024 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.585023045539856 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9915685653686523 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17965859174728394 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21881742775440216 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _step‚795 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16628967225551605 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3277834355831146 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166567325592 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 jet/accuracy_5_of_7‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.02131783217191696 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 *jet/accuracy_1_of_2‚0.7370837330818176 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3543851673603058 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5088329911231995 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17868895828723907 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19797030091285706 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4344947636127472 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37586650252342224 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.580926239490509 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16592943668365479 particle/accuracy_4_of_5‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18009687960147858 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29526567459106445 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 )jet/accuracy_2_of_4‚0.369784951210022 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15678198635578156 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4782078266143799 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 (particle/accuracy‚0.8538618683815002 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24916110932826996 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1884889453649521 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9905454516410828 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48355263471603394 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5633686780929565 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.516905665397644 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4583333134651184 particle/accuracy_3_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9924482107162476 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_2_of_2‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4630506932735443 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16563262045383453 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.42902523279190063 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.196675643324852 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9460893273353577 -jet/accuracy_5_of_5‚0.0026881720405071974 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5101527571678162 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3913942873477936 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014153439551591873 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23989899456501007 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6510269641876221 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3055555522441864 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 particle/accuracy_5_of_6‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4504068195819855 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 jet/accuracy_4_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46616673469543457 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.013207548297941685 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10768849402666092 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3516278862953186 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16261771321296692 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5097315907478333 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42318424582481384 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 particle/accuracy_4_of_4‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5271851420402527 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9965744018554688 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9915668964385986 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26559048891067505 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 particle/accuracy_1_of_2‚1 jet/accuracy_6_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1571839153766632 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4032737910747528 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.388552188873291 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11149377375841141 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5443349480628967 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +particle/specificity‚0.9507817625999451 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 *jet/accuracy_2_of_6‚0.4551282227039337 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4907848536968231 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4502054750919342 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_6‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5486898422241211 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29976755380630493 ,jet/accuracy_4_of_5‚0.025179212912917137 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5801843404769897 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3261905014514923 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5279174447059631 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46888843178749084 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6225729584693909 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 _runtime‚7571.151433863 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42088350653648376 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5555084943771362 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17500722408294678 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04054022952914238 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5017039775848389 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5009503364562988 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4495305120944977 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08773097395896912 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.445242315530777 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.39726522564888 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024872079491615295 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005885560996830463 ( Purity/2FRt4SRqqt0FBt/FRt_purity‚0.5 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24727505445480347 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8131499290466309 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23468345403671265 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18924285471439362 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.47531822323799133 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919971227645874 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15589384734630585 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.38398686051368713 jet/accuracy_2_of_7‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02825211174786091 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19551584124565125 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4582873582839966 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18957103788852692 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2805555760860443 particle/accuracy_2_of_6‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.47518524527549744 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.029051460325717926 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47053858637809753 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15016891062259674 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2410714328289032 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_5‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5375174880027771 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4797461926937103 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9901708960533142 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983689785003662 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ( Purity/2FRt4SRqqt*FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.474642813205719 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02211165614426136 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5028092265129089 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 jet/accuracy_3_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40086838603019714 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46888843178749084 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.997774064540863 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45603445172309875 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4704861640930176 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23055203258991241 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2465517222881317 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3759258985519409 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_2_of_3‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40588653087615967 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26919281482696533 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41296303272247314 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 +jet/accuracy_2_of_3‚0.23717811703681946 particle/accuracy_6_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9231677651405334 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3684028089046478 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19870202243328094 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.576785683631897 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5264991521835327 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864625334739685 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.42635664343833923 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1û8  *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 particle/accuracy_7_of_7‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.42279699444770813 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.0416666679084301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1608186960220337 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5044039487838745 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 jet/accuracy_1_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5702381730079651 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895066618919373 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12643678486347198 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5115740895271301 ,jet/accuracy_4_of_6‚0.057692307978868484 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2826535105705261 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994331002235413 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3055555522441864 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2545045018196106 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 +particle/sensitivity‚0.3301701843738556 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894058704376221 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727›%©ß’¹$‚(:‡ ¸„äÔæñ  gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.193' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™~K|º$‚(:ñ ¸„äÔïÚº  disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.sda2.out‚0& disk.nvme0c0n1.out‚50866.322265625 disk.drbd2265.out‚0* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 349.3984375 disk.sda.in‚ 0.1953125 disk.sda.out‚ 349.3984375) proc.memory.availableMB‚ 2748.03515625% disk./.usageGB‚246.67951583862305 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.rbd1.out‚34.22021484375$ disk.nvme0n1.out‚50866.322265625 disk.dm-0.in‚ 1187.62109375 cpu‚25.36460461937315 disk.sda1.out‚0! disk.dm-0.out‚50866.322265625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.out‚ 349.3984375$ memory_percent‚83.22732448577881" proc.memory.rssMB‚ 2810.9296875 network.sent‚ 2.3524418e+07 network.recv‚ 2.792415e+06* disk./.usagePercent‚16.554869285829913 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0\¼M»$‚(:… Ç„äÔ™•Ÿ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.438 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wú ²~¼$‚(:ó Ç„äÔÿÖŠ  disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 34.283203125 disk.sda3.out‚ 350.0859375 cpu‚25.36398751686314 disk.sda2.in‚0( disk.nvme0c0n1.out‚50918.86083984375 disk.sda.out‚ 350.0859375 network.recv‚ 2.799297e+06 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.rbd0.out‚0( proc.memory.availableMB‚ 2748.0390625* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17 disk.sda1.in‚0& disk.nvme0n1.out‚50918.86083984375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 network.sent‚ 2.3648726e+07* disk./.usagePercent‚16.556027975929887 disk.sda1.out‚0 disk.drbd2265.out‚0# memory_percent‚83.2273006439209# disk.nvme0c0n1.in‚ 1187.62109375# disk.dm-0.out‚50918.86083984375 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0% disk./.usageGB‚246.69678115844727 disk.dm-1.out‚ 350.0859375 disk.drbd2266.in‚0 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2810.9296875ik„¼2½$- _wandbruntime‚7591  _runtime‚7591P¶Áóƾ$À _runtime‚ 7591.15001881 _step‚796 & lr-AdamW/pg1‚3.148986344597322e-05 & lr-AdamW/pg2‚3.148986344597322e-05 trainer/global_step‚9699 % _timestamp‚1.7884124881625757e+09¶x„fË¿$Å % _timestamp‚1.7884124881625757e+09 _runtime‚ 7591.15001881 _step‚796 & lr-AdamW/pg1‚3.148986344597322e-05 & lr-AdamW/pg2‚3.148986344597322e-05 trainer/global_step‚9699œ†¿2À$- _wandbruntime‚7591  _runtime‚7591ãú„ç'Á$áO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573338860993024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8135155169280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129016712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7591.151195791 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8135155169280 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141409022 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚550373804602368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚797 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183344196 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558479675833630ÇïoAì'Â$æO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚797 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183344196 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558479675833630 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573338860993024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8135155169280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129016712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7591.151195791 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8135155169280 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141409022 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚550373804602368nñŒõ2Ã$- _wandbruntime‚7591  _runtime‚75911ŽCU„Ä$ëM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617183831075840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183362650 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558535396399362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚550428708732928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141423036 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚565260076865536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141423038 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46507866 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129028908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82521484 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129029352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573396707838464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82521486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚550428725772288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573396033459712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183382714 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324785686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617183807135744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141423036 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324785686 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚565260058862592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135974597120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625718634697216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8534803621376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8106670627074 VHDeviceStatsMonitor.on_train_batch_´ >mend/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46507424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9699 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚565260076865536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8136630972928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8534635816448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8106016189110 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚558534724922038 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625718442952192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8135974597120 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183362650 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324805752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141423038 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573396707838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚565260058862592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884124888502648e+09 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8136630972928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8534635816448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8106016189110 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚558534724922038 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625718442952192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8135974597120 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183362650 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324805752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141423038 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573396707838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚565260058862592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884124888502648e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617183831075840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183362650 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558535396399362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚550428708732928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141423036 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚565260076865536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141423038 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46507866 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129028908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82521484 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129029352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573396707838464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82521486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚550428725772288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573396033459712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183382714 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324785686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617183807135744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141423036 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324785686 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚565260058862592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135974597120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625718634697216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8534803621376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8106670627074 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46507424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9699 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚565260076865536žòh'*2Æ$- _wandbruntime‚7591  _runtime‚7591Tài_Ç$Ù _runtime‚7591.839506551 /loss/FRt2/detection_loss‚2.0723953552421035 0loss/SRqqt1/detection_loss‚2.497354140204768 &loss/total_loss‚41.693954843756515 /loss/FBt1/detection_loss‚3.2378225922584534 epoch‚8 1loss/FBt3/assignment_loss‚0.11039941725720406 /loss/FRt1/detection_loss‚3.0232726242393255 1loss/SRqqt4/detection_loss‚1.7005918557169262 0loss/FRt1/assignment_loss‚3.3343451507389545 2loss/SRqqt2/assignment_loss‚1.5195691461427783 1loss/SRqqt2/detection_loss‚1.6691081037326736 1loss/SRqqt3/detection_loss‚1.0622871473638549 .loss/FBt4/detection_loss‚4.096676141023636 _step‚799 0loss/FRt3/assignment_loss‚1.1472969742144568 $ _timestamp‚1.788412488852142e+09 /loss/FRt2/assignment_loss‚2.837784000805446 2loss/FBt1/assignment_loss‚0.007342786598019302 /loss/FRt4/detection_loss‚2.4915296173095705 /loss/FBt2/detection_loss‚1.8784489860901465 2loss/SRqqt1/assignment_loss‚1.7820233215517816 1loss/SRqqt3/assignment_loss‚1.051609337310462 2loss/FBt2/assignment_loss‚0.054108003062538046 /loss/FRt3/detection_loss‚1.6765130051111772 .loss/FBt3/detection_loss‚0.562677168212601 trainer/global_step‚9699 /loss/FRt4/assignment_loss‚2.485356696844101 2loss/SRqqt4/assignment_loss‚1.2980628635182425 1loss/FBt4/assignment_loss‚0.09738040920728963ò­ÀdÈ$Þ /loss/FBt1/detection_loss‚3.2378225922584534 epoch‚8 1loss/FBt3/assignment_loss‚0.11039941725720406 /loss/FRt1/detection_loss‚3.0232726242393255 1loss/SRqqt4/detection_loss‚1.7005918557169262 0loss/FRt1/assignment_loss‚3.3343451507389545 2loss/SRqqt2/assignment_loss‚1.5195691461427783 1loss/SRqqt2/detection_loss‚1.6691081037326736 1loss/SRqqt3/detection_loss‚1.0622871473638549 .loss/FBt4/detection_loss‚4.096676141023636 _step‚799 0loss/FRt3/assignment_loss‚1.1472969742144568 $ _timestamp‚1.788412488852142e+09 /loss/FRt2/assignment_loss‚2.837784000805446 2loss/FBt1/assignment_loss‚0.007342786598019302 /loss/FRt4/detection_loss‚2.4915296173095705 /loss/FBt2/detection_loss‚1.8784489860901465 2loss/SRqqt1/assignment_loss‚1.7820233215517816 1loss/SRqqt3/assignment_loss‚1.051609337310462 2loss/FBt2/assignment_loss‚0.054108003062538046 /loss/FRt3/detection_loss‚1.6765130051111772 .loss/FBt3/detection_loss‚0.562677168212601 trainer/global_step‚9699 /loss/FRt4/assignment_loss‚2.485356696844101 2loss/SRqqt4/assignment_loss‚1.2980628635182425 1loss/FBt4/assignment_loss‚0.09738040920728963 _runtime‚7591.839506551 /loss/FRt2/detection_loss‚2.0723953552421035 0loss/SRqqt1/detection_loss‚2.497354140204768 &loss/total_loss‚41.693954843756515Ÿ¿!`É$‚(:’ Ö„äÔÙŽô gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.338' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.867599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚084óɃÊ$‚(:ø Ö„äԧ܃ & disk.nvme0n1.out‚51010.92529296875 disk.sda3.out‚ 350.72265625* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17* disk./.usagePercent‚16.557679992548337 disk.sda1.in‚0$ memory_percent‚83.22904109954834 cpu‚25.36340340475984 disk.drbd1716.in‚0 disk.dm-1.out‚ 350.72265625 disk.drbd2265.out‚0 disk.sda.out‚ 350.72265625 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1187.62109375# disk.dm-0.out‚51010.92529296875( disk.nvme0c0n1.out‚51010.92529296875) proc.memory.availableMB‚ 2747.75390625" proc.memory.rssMB‚ 2810.9296875 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.drbd2266.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 network.recv‚ 2.804072e+06% disk./.usageGB‚246.72139739990234 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 34.369140625 disk.drbd2265.in‚0 network.sent‚ 2.3654766e+07/üî’Ë$‚(:‡ å„äÔºŠç gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.038' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Ï_è„Ì$‚(:ù å„äÔ÷Èï  disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd1.out‚ 34.431640625# disk.dm-0.out‚51103.70556640625 disk.sda.out‚ 351.59765625" proc.memory.rssMB‚ 2810.9296875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 351.59765625$ memory_percent‚83.22746753692627) proc.memory.availableMB‚ 2748.01171875 network.sent‚ 2.3764094e+07 network.recv‚ 2.811184e+06 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.out‚0 cpu‚25.362743115833528& disk.nvme0n1.out‚51103.70556640625 disk.dm-1.out‚ 351.59765625% disk./.usageGB‚246.72692108154297 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.drbd2265.in‚0* proc.memory.percent‚17.156553268432617 disk.rbd0.out‚0 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚51103.70556640625 disk.dm-1.in‚ 0.1953125* disk./.usagePercent‚16.558050691457918 disk.drbd2080.out‚0fÆ2Í$- _wandbruntime‚7628  _runtime‚7628/‘9åÉÎ$à _step‚800 ' lr-AdamW/pg1‚3.0236663918835243e-05 ' lr-AdamW/pg2‚3.0236663918835243e-05 trainer/global_step‚9749 % _timestamp‚1.7884125251119924e+09 _runtime‚7628.099435508ŠÂ~ÀÎÏ$È _step‚800 ' lr-AdamW/pg1‚3.0236663918835243e-05 ' lr-AdamW/pg2‚3.0236663918835243e-05 trainer/global_step‚9749 % _timestamp‚1.7884125251119924e+09 _runtime‚7628.099435508 pNì 2Ð$- _wandbruntime‚7628  _runtime‚7628fد&æ'Ñ$àO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184290286 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142109820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚568021357766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46727406 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8576814314496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8146689531310 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8146035093554 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82921718 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8176797337600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚ 7628.10035853 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129649122 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628781531535872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚576198173106688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚576197498741248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620204693281280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8176140975104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8176797337600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326400108 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129649574 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82921716 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8176140975104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9749 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184270248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚576198173106688 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568021357766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚561265717701038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184290286 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46727856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚801 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚568021375769088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚561265046223922 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8146689531310 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requesöû©dÃ$ted_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8146035093554 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82921718 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8176797337600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚ 7628.10035853 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129649122 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628781531535872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚576198173106688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚576197498741248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620204693281280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8176140975104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8176797337600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326400108 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129649574 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82921716 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8176140975104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9749 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184270248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚576198173106688 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568021357766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚561265717701038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184290286 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46727856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚801 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚568021375769088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚561265046223922 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129661702 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561321438266770 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚568077710804992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46731818 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142123838 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8176960402944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8176960402944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568077728807936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚568077710804992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620266344018944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8146851528518 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚576254671207936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46732277 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553173915260928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8147505966482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7628.790220354 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326432604 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184288702 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129662163 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚576255345586688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8177616778752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432604 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553173932300288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326412538 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184308766 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82929886 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8577657529856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884125258022277e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8577481336320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142123836 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561320766789446 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620266367959040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142123838 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184308766 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184288702 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8177616778752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0-vKö&Õ$ðM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561320766789446 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620266367959040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142123838 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184308766 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184288702 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8177616778752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚576255345586688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129661702 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561321438266770 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚568077710804992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46731818 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142123838 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8176960402944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8176960402944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568077728807936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚568077710804992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620266344018944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8146851528518 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚576254671207936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46732277 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553173915260928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8147505966482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7628.790220354 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326432604 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184288702 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129662163 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚576255345586688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8177616778752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432604 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553173932300288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326412538 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184308766 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82929886 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8577657529856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884125258022277e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8577481336320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142123836 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521¢˜ Þµ2Ö$- _wandbruntime‚7628  _runtime‚7628xg^×$Ø 2loss/SRqqt3/assignment_loss‚1.2247620604017324 1loss/FBt3/assignment_loss‚0.23570823837951463 /loss/FBt2/detection_loss‚2.0873724073171616 0loss/FRt2/assignment_loss‚2.7359977094632275 /loss/FRt2/detection_loss‚1.9990104551745556 0loss/SRqqt2/detection_loss‚1.562847796133009 /loss/FBt1/detection_loss‚4.5309076110521955 % _timestamp‚1.7884125258038836e+09 1loss/FBt1/assignment_loss‚0.02372452011331916 1loss/FBt2/assignment_loss‚0.14208098239760147 0loss/FBt4/assignment_loss‚0.2236873876924316 trainer/global_step‚9749 /loss/FRt4/assignment_loss‚2.641932194297378 /loss/FBt3/detection_loss‚0.5664939477988894 epoch‚8 .loss/FRt3/detection_loss‚1.592722175010879 1loss/SRqqt3/detection_loss‚1.1849398822768737 1loss/SRqqt4/detection_loss‚1.9269128860533238 .loss/FBt4/detection_loss‚3.451479216416677 _step‚803 /loss/FRt3/assignment_loss‚1.217858618923596 2loss/SRqqt2/assignment_loss‚1.2083081916069327 /loss/FRt1/detection_loss‚2.7556189612338415 &loss/total_loss‚42.906204533184294 0loss/FRt1/assignment_loss‚3.4922127378614327 2loss/SRqqt4/assignment_loss‚1.1623682672157885 /loss/FRt4/detection_loss‚2.6028727667557225 1loss/SRqqt1/detection_loss‚2.7759908074918003 _runtime‚7628.791248664 2loss/SRqqt1/assignment_loss‚1.5603947121164072Bû"cØ$Ý /loss/FBt1/detection_loss‚4.5309076110521955 % _timestamp‚1.7884125258038836e+09 1loss/FBt1/assignment_loss‚0.02372452011331916 1loss/FBt2/assignment_loss‚0.14208098239760147 0loss/FBt4/assignment_loss‚0.2236873876924316 trainer/global_step‚9749 /loss/FRt4/assignment_loss‚2.641932194297378 /loss/FBt3/detection_loss‚0.5664939477988894 epoch‚8 .loss/FRt3/detection_loss‚1.592722175010879 1loss/SRqqt3/detection_loss‚1.1849398822768737 1loss/SRqqt4/detection_loss‚1.9269128860533238 .loss/FBt4/detection_loss‚3.451479216416677 _step‚803 /loss/FRt3/assignment_loss‚1.217858618923596 2loss/SRqqt2/assignment_loss‚1.2083081916069327 /loss/FRt1/detection_loss‚2.7556189612338415 &loss/total_loss‚42.906204533184294 0loss/FRt1/assignment_loss‚3.4922127378614327 2loss/SRqqt4/assignment_loss‚1.1623682672157885 /loss/FRt4/detection_loss‚2.6028727667557225 1loss/SRqqt1/detection_loss‚2.7759908074918003 _runtime‚7628.791248664 2loss/SRqqt1/assignment_loss‚1.5603947121164072 2loss/SRqqt3/assignment_loss‚1.2247620604017324 1loss/FBt3/assignment_loss‚0.23570823837951463 /loss/FBt2/detection_loss‚2.0873724073171616 0loss/FRt2/assignment_loss‚2.7359977094632275 /loss/FRt2/detection_loss‚1.9990104551745556 0loss/SRqqt2/detection_loss‚1.562847796133009£å¾ Ù$‚(:’ ô„äÔ°¤› gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.631' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.126200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01H£³UÚ$‚(:ó ô„äÔ¹·›  disk.sda3.out‚ 352.4765625 disk.dm-1.in‚ 0.1953125[Ñ”8) disk.drbd2266.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2080.out‚0& disk.nvme0n1.out‚51206.80810546875 cpu‚25.362102566793208 disk.drbd1716.in‚0 disk.drbd1716.out‚0% disk./.usageGB‚246.76275634765625 disk.dm-0.in‚ 1187.62109375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚ 34.431640625# memory_percent‚83.2277774810791) disk./.usagePercent‚16.56045562623457 disk.sda1.out‚0" proc.memory.rssMB‚ 2810.9296875 network.sent‚ 2.3769333e+07# disk.dm-0.out‚51206.80810546875 disk.drbd2080.in‚0* proc.memory.percent‚17.156553268432617 disk.dm-1.out‚ 352.4765625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 network.recv‚ 2.815055e+06 disk.sda.out‚ 352.4765625 disk.drbd2266.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚51206.80810546875( proc.memory.availableMB‚ 2747.9609375 proc.cpu.threads‚17~e`’Û$‚(:‡ ƒ…äÔöŽ‹ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Ð¦„Ü$‚(:ù ƒ…äÔ‚ßí* disk./.usagePercent‚16.562558216113285# disk.nvme0c0n1.in‚ 1187.62109375( disk.nvme0c0n1.out‚51269.56982421875 disk.rbd1.out‚ 34.4638671875! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚51269.56982421875# disk.dm-0.out‚51269.56982421875 disk.sda3.out‚ 352.97265625 network.sent‚ 2.3878087e+07 disk.rbd1.in‚0 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2810.9296875 disk.sda2.in‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0 disk.rbd0.in‚0$ memory_percent‚83.22947025299072 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17% disk./.usageGB‚246.79408645629883 disk.sda1.in‚0 disk.dm-1.out‚ 352.97265625 disk.sda.out‚ 352.97265625 disk.rbd0.out‚0) proc.memory.availableMB‚ 2747.68359375 network.recv‚ 2.821972e+06 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 cpu‚25.36150191248554lI”£’Ý$‚(:‡ ’…äÔëËœ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.643' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1286 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0côbƒÞ$‚(:ø ’…äÔ…φ ( proc.memory.availableMB‚ 2747.9296875* proc.memory.percent‚17.156553268432617 cpu‚25.36089297273798 proc.cpu.threads‚17 disk.sda3.in‚ 0.1953125 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 353.61328125 disk.drbd1716.out‚0 network.recv‚ 2.826466e+06 disk.sda1.out‚0 disk.sda.in‚ 0.1953125 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚51363.60888671875 disk.drbd2265.in‚0 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2810.9296875 disk.sda3.out‚ 353.61328125 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 34.4638671875 disk.drbd2265.out‚0% disk./.usageGB‚246.83429718017578 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375( disk.nvme0c0n1.out‚51363.60888671875 disk.sda.out‚ 353.61328125 network.sent‚ 2.3883935e+07* disk./.usagePercent‚16.565256791531713! disk.nvme0n1.in‚ 1187.62109375 disk.sda1.in‚0& disk.nvme0n1.out‚51363.60888671875$ memory_percent‚83.22796821594238:Y%2ß$- _wandbruntime‚7665  _runtime‚7665LC.Éà$à _runtime‚7665.134894931 _step‚804 ' lr-AdamW/pg1‚2.9005824063113097e-05 ' lr-AdamW/pg2‚2.9005824063113097e-05 trainer/global_step‚9799 % _timestamp‚1.7884125621474435e+09 áõðÎá$È _runtime‚7665.134894931 _step‚804 ' lr-AdamW/pg1‚2.9005824063113097e-05 ' lr-AdamW/pg2‚2.9005824063113097e-05 trainer/global_step‚9799 % _timestamp‚1.7884125621474435e+09¤~Ïé)2â$- _wandbruntime‚7665  _runtime‚7665ö5$aç'ã$áO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚623287230164480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46952950 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555864234697728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8217126780928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83330116 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564051759568446 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884125621481519e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328006920 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚579056810854912 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚9799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570839009708544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142810622 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579056136489472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8217783143424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8619480467968 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130283066 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚805 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8217126780928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328026960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186870432962 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚7665.135821559 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚570839027711488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142810622 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185196300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚579056136489472 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130283495 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚631906710632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8619646189056 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185196300 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185216338 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8187524870718Y10pì'ä$æO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555864234697728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8217126780928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83330116 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564051759568446 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884125621481519e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328006920 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚579056810854912 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚9799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570839009708544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142810622 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579056136489472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8217783143424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8619480467968 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130283066 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚805 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8217126780928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328026960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186870432962 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚7665.135821559 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚570839027711488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142810622 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185196300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚579056136489472 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130283495 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚631906710632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8619646189056 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185196300 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185216338 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8187524870718 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚623287230164480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46952950 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116¥…ƒ32å$- _wandbruntime‚7665  _runtime‚7665*œ¯Qæ$ëM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185234818 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚570895362747392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185214754 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83338284 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚9799 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564106808656854 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚570895362747392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631969223220224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328059456 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚631969410770944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059456 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130296198 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579113983334912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚623348880902144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8217946208768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130295759 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328039390 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142824636 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564107480134178 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579113308956160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83338286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46957912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8620505928704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185234818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8188341305890 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8217946208768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8218602584576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8218602584576 C;DeviceStatsMonitor.on_train_batcÂÖa  h_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570895380750336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555919138828288 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185214754 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187686867926 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8620342318080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844VÏéö&ç$ðM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328059456 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚631969410770944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059456 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130296198 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579113983334912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚623348880902144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8217946208768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130295759 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328039390 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142824636 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564107480134178 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579113308956160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83338286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46957912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8620505928704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185234818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8188341305890 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8217946208768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8218602584576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8218602584576 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570895380750336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555919138828288 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185214754 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187686867926 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8620342318080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185234818 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚570895362747392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185214754 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83338284 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚9799 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564106808656854 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚570895362747392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631969223220224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844¦úÏM2è$- _wandbruntime‚7665  _runtime‚7665”£¨²Yé$Ó /loss/FRt1/assignment_loss‚3.509244521458944 2loss/SRqqt2/assignment_loss‚1.0823738692471614 0loss/FBt2/assignment_loss‚0.3184401042420756 1loss/FBt3/assignment_loss‚0.15690589576154454 %loss/total_loss‚42.23436781570905 trainer/global_step‚9799 _step‚807 0loss/SRqqt2/detection_loss‚2.078274366947321 1loss/SRqqt4/detection_loss‚1.9284518467652243 epoch‚8 2loss/SRqqt1/assignment_loss‚1.4417059693485499 .loss/FBt2/detection_loss‚2.200112061067061 0loss/FRt3/assignment_loss‚0.9412670825041977 /loss/FRt2/detection_loss‚2.3521546299916376 /loss/FRt4/detection_loss‚2.1452379995584487 _runtime‚7665.826457559 0loss/FRt2/assignment_loss‚2.9100633336276545 .loss/FRt1/detection_loss‚2.966034948825836 .loss/FRt3/detection_loss‚1.638781386738022 /loss/FBt3/detection_loss‚0.6192647570708726 /loss/FBt4/detection_loss‚2.7766867876052856 % _timestamp‚1.7884125628390937e+09 0loss/FRt4/assignment_loss‚3.1625295758247374 1loss/SRqqt3/detection_loss‚1.0798263012341434 2loss/SRqqt3/assignment_loss‚0.8414095247665706 0loss/FBt1/assignment_loss‚0.9243032081983984 1loss/SRqqt1/detection_loss‚2.7495142022768655 2loss/SRqqt4/assignment_loss‚1.4773239989867868 0loss/FBt4/assignment_loss‚0.2934988667955622 -loss/FBt1/detection_loss‚2.64096257686615¾?â}^ê$Ø _step‚807 0loss/SRqqt2/detection_loss‚2.078274366947321 1loss/SRqqt4/detection_loss‚1.9284518467652243 epoch‚8 2loss/SRqqt1/assignment_loss‚1.4417059693485499 .loss/FBt2/detection_loss‚2.200112061067061 0loss/FRt3/assignment_loss‚0.9412670825041977 /loss/FRt2/detection_loss‚2.3521546299916376 /loss/FRt4/detection_loss‚2.1452379995584487 _runtime‚7665.826457559 0loss/FRt2/assignment_loss‚2.9100633336276545 .loss/FRt1/detection_loss‚2.966034948825836 .loss/FRt3/detection_loss‚1.638781386738022 /loss/FBt3/detection_loss‚0.6192647570708726 /loss/FBt4/detection_loss‚2.7766867876052856 % _timestamp‚1.7884125628390937e+09 0loss/FRt4/assignment_loss‚3.1625295758247374 1loss/SRqqt3/detection_loss‚1.0798263012341434 2loss/SRqqt3/assignment_loss‚0.8414095247665706 0loss/FBt1/assignment_loss‚0.9243032081983984 1loss/SRqqt1/detection_loss‚2.7495142022768655 2loss/SRqqt4/assignment_loss‚1.4773239989867868 0loss/FBt4/assignment_loss‚0.2934988667955622 -loss/FBt1/detection_loss‚2.64096257686615 /loss/FRt1/assignment_loss‚3.509244521458944 2loss/SRqqt2/assignment_loss‚1.0823738692471614 0loss/FBt2/assignment_loss‚0.3184401042420756 1loss/FBt3/assignment_loss‚0.15690589576154454 %loss/total_loss‚42.23436781570905 trainer/global_step‚9799§ȶüˆë$‚(:’ ¡…äÔÖáî gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.269' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.653800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q …ì$‚(:ú ¡…äÔíè“  disk.sda3.out‚ 354.05078125 disk.rbd1.in‚0* proc.memory.percent‚17.156553268432617* disk./.usagePercent‚16.567397526533032 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2810.9296875% disk./.usageGB‚246.86619567871094 disk.dm-0.in‚ 1187.62109375 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0$ memory_percent‚83.22818279266357# disk.dm-0.out‚51445.67041015625 disk.dm-1.out‚ 354.05078125 disk.sda.out‚ 354.05078125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚ 2.3890291e+07 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚34.52685546875 cpu‚25.36028699317411 network.recv‚ 2.830818e+06 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚51445.67041015625) proc.memory.availableMB‚ 2747.89453125 disk.drbd2265.in‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚51445.67041015625hì’í$‚(:‡ °…äÔù­¤ gpu.0.gpu‚41.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eðsî$‚(:è °…äÔñÅŽ ! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.dm-1.out‚ 354.984375$ memory_percent‚83.22703838348389 disk.drbd2080.out‚0$ disk.nvme0c0n1.out‚ 51568.7109375" disk.nvme0n1.out‚ 51568.7109375 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.dm-0.out‚ 51568.7109375" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0* disk./.usagePercent‚16.569355728162435 disk.sda.out‚ 354.984375 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 network.sent‚ 2.4000649e+07 network.recv‚ 2.837381e+06$ disk./.usageGB‚246.8953742980957 disk.rbd0.out‚0* proc.memory.percent‚17.156553268432617 disk.sda3.out‚ 354.984375 disk.sda2.in‚0 disk.drbd2265.in‚0) proc.memory.availableMB‚ 2748.08203125 disk.rbd0.in‚0 cpu‚25.359677502719613 disk.drbd2266.in‚0 disk.rbd1.out‚34.52685546875{¤ž2ï$- _wandbruntime‚7702  _runtime‚7702éóÃ}Çð$Á & lr-AdamW/pg1‚2.779760038398797e-05 & lr-AdamW/pg2‚2.779760038398797e-05 trainer/global_step‚9849 % _timestamp‚1.7884125991793077e+09 _runtime‚7702.166729967 _step‚808Îb sÌñ$Æ _step‚808 & lr-AdamW/pg1‚2.779760038398797e-05 & lr-AdamW/pg2‚2.779760038398797e-05 trainer/global_step‚9849 % _timestamp‚1.7884125991793077e+09 _runtime‚7702.166729967¨„Œûx2ò$- _wandbruntime‚7702  _runtime‚7702%Rùèç'ó$áO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83738516 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8662287240192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186122352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8662446669824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8227705772370 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8258112586752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573656661650944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573656679653888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130918425 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚9849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329633772 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8258768949248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635032054287872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8228360210126 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130917985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚558609441225728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚626369767047680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚581915448603136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884125991800683e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143511422 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83738518 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635032237657600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573656661650944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚809 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143511420 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚581915448603136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626369790987776 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581914774237696 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186122352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8258768949248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143511422 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚566837801435854 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186142390 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚581914774237696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚566837129958738 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329653812 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚558609424186368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573656679653888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47179907.í²Œô$æO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8258112586752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47179469 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143511420 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚558609424186368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573656679653888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47179907 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83738516 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8662287240192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186122352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8662446669824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8227705772370 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8258112586752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573656661650944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_po5ÈÔòÙol.allocated‚573656679653888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130918425 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚9849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329633772 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8258768949248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635032054287872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8228360210126 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130917985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚558609441225728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚626369767047680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚581915448603136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884125991800683e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143511422 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83738518 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635032237657600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573656661650944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚809 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143511420 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚581915448603136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626369790987776 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581914774237696 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186122352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8258768949248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143511422 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚566837801435854 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186142390 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚581914774237696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚566837129958738 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329653812 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686308 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130931116 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130930694 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8258932014592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626431417785344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8228522207334 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143525436 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581971946704384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚573713014689792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚422 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚558664328316928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚566893522001586 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566892850524262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666242 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚558664345356288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143525438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884125998721547e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚581972621083136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8663312858624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47184430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47184010 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329686308 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329666242 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143525438 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635094569130496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186140806 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚573713032692736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8258932014592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944Æã2ö&÷$ðM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566892850524262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666242 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚558664345356288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143525438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884125998721547e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚581972621083136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8663312858624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47184430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47184010 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329686308 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329666242 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143525438 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635094569130496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186140806 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚573713032692736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8258932014592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚573713014689792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚573713032692736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686308 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130931116 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130930694 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8258932014592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626431417785344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8228522207334 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143525436 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581971946704384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚573713014689792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚422 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚558664328316928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚566893522001586ªý¹Ý2ø$- _wandbruntime‚7702  _runtime‚77027ªÛWù$Ñ 2loss/SRqqt2/assignment_loss‚1.4555868801533007 1loss/FBt4/assignment_loss‚0.12869258797062294 1loss/SRqqt2/detection_loss‚1.7547406232824512 _step‚811 trainer/global_step‚9849 0loss/FRt1/assignment_loss‚3.1384055773417154 0loss/FRt4/assignment_loss‚2.4805717719925773 .loss/FRt3/detection_loss‚1.743893743516543 1loss/SRqqt3/detection_loss‚1.0308540345480044 .loss/FBt4/detection_loss‚2.894891245024545 /loss/FRt2/assignment_loss‚2.882096943355376 2loss/SRqqt3/assignment_loss‚0.7812232893513282 epoch‚8 /loss/FRt2/detection_loss‚2.0474446616043003 0loss/SRqqt1/detection_loss‚2.359591044259794 .loss/FBt2/detection_loss‚2.103237122297287 1loss/SRqqt1/assignment_loss‚1.636919050223448 1loss/SRqqt4/detection_loss‚1.9087149034789268 /loss/FBt3/detection_loss‚0.7361437661032116 $loss/total_loss‚39.9993137083048 0loss/FRt3/assignment_loss‚1.2904561773324623 2loss/SRqqt4/assignment_loss‚1.2703664961488956 1loss/FBt3/assignment_loss‚0.09649485220046587 _runtime‚7702.861144902 0loss/FBt2/assignment_loss‚0.1936333989026025 .loss/FBt1/detection_loss‚2.602351874113083 $ _timestamp‚1.788412599873757e+09 1loss/FBt1/assignment_loss‚0.25654645882812993 .loss/FRt1/detection_loss‚2.752479656537374 .loss/FRt4/detection_loss‚2.453977549738354ó~e¢\ú$Ö 2loss/SRqqt2/assignment_loss‚1.4555868801533007 1loss/FBt4/assignment_loss‚0.12869258797062294 1loss/SRqqt2/detection_loss‚1.7547406232824512 _step‚811 trainer/global_step‚9849 0loss/FRt1/assignment_loss‚3.1384055773417154 0loss/FRt4/assignment_loss‚2.4805717719925773 .loss/FRt3/detection_loss‚1.743893743516543 1loss/SRqqt3/detection_loss‚1.0308540345480044 .loss/FBt4/detection_loss‚2.894891245024545 /loss/FRt2/assignment_loss‚2.882096943355376 2loss/SRqqt3/assignment_loss‚0.7812232893513282 epoch‚8 /loss/FRt2/detection_loss‚2.0474446616043003 0loss/SRqqt1/detection_loss‚2.359591044259794 .loss/FBt2/detection_loss‚2.103237122297287 1loss/SRqqt1/assignment_loss‚1.636919050223448 1loss/SRqqt4/detection_loss‚1.9087149034789268 /loss/FBt3/detection_loss‚0.7361437661032116 $loss/total_loss‚39.9993137083048 0loss/FRt3/assignment_loss‚1.2904561773324623 2loss/SRqqt4/assignment_loss‚1.2703664961488956 1loss/FBt3/assignment_loss‚0.09649485220046587 _runtime‚7702.861144902 0loss/FBt2/assignment_loss‚0.1936333989026025 .loss/FBt1/detection_loss‚2.602351874113083 $ _timestamp‚1.788412599873757e+09 1loss/FBt1/assignment_loss‚0.25654645882812993 .loss/FRt1/detection_loss‚2.752479656537374 .loss/FRt4/detection_loss‚2.453977549738354«šœ˜’û$‚(:‡ ¿…äÔ’ýª gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0> Ò´qü$‚(:æ ¿…äÔ›Ò—  disk.sda3.out‚ 355.66796875 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 51637.421875" proc.memory.rssMB‚ 2810.9296875! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.rbd1.out‚ 34.58984375 disk.drbd2266.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 355.66796875 network.recv‚ 2.841462e+06 disk.dm-0.out‚ 51637.421875 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125$ memory_percent‚83.22367668151855 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.1953125! disk.nvme0n1.out‚ 51637.421875 disk.sda.out‚ 355.66796875( proc.memory.availableMB‚ 2748.6328125* proc.memory.percent‚17.156553268432617 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.sda2.out‚0 disk.rbd0.in‚0 network.sent‚ 2.4004483e+07% disk./.usageGB‚246.91947174072266 disk.rbd1.in‚0 cpu‚25.35907659995893 proc.cpu.threads‚17) disk./.usagePercent‚16.57097292775623 disk.drbd1716.in‚0 disk.drbd1716.out‚0m¢^ý$‚(:’ Î…äԛŠgpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.319' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.263799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ—…~þ$‚(:ó Î…äÔ£Û¢  disk.sda.in‚ 0.1953125 disk.sda3.out‚ 356.6171875 disk.drbd2080.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 2748.62890625% disk./.usageGB‚246.94108963012695 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd1.out‚ 34.58984375! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚51706.43310546875 network.recv‚ 2.848067e+06* proc.memory.percent‚17.156553268432617) disk./.usagePercent‚16.57242372245277 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚51706.43310546875 disk.drbd2266.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.4112939e+07 disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 356.6171875 disk.sda.out‚ 356.6171875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2266.out‚0# disk.nvme0c0n1.in‚ 1187.62109375( disk.nvme0c0n1.out‚51706.43310546875$ memory_percent‚83.22370052337646 cpu‚25.35847778879005 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2810.9296875 disk.drbd1716.in‚0 disk.drbd1716.out‚0àDá2ÿ$- _wandbruntime‚7738  _runtime‚7738£íDˆ~€%à ' lr-AdamW/pg1‚2.6612244673463668e-05 ' lr-AdamW/pg2‚2.6612244673463668e-05 trainer/global_step‚9899 % _timàxéKestamp‚1.7884126357645202e+09 _runtime‚7738.751962052 _step‚812j—¼Î%È _runtime‚7738.751962052 _step‚812 ' lr-AdamW/pg1‚2.6612244673463668e-05 ' lr-AdamW/pg2‚2.6612244673463668e-05 trainer/global_step‚9899 % _timestamp‚1.7884126357645202e+09¬©ádß2‚%- _wandbruntime‚7738  _runtime‚7738:³æç'ƒ%áO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8705367660032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331260624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584773411985920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131554294 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚629452327870976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576474331596288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8299098392576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584773411985920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144212220 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638157520549888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569623843303262 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569623171826146 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144212222 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576474313593344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8268541111778 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561354647753728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚9899 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187048404 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576474313593344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚813 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331280664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8299754755072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187068442 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144212220 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187048404 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144212222 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629452303930880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584774086351360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7738.752932964 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8299098392576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584773411985920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131554294 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚629452327870976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576474331596288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8299098392576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584773411985920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144212220 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638157520549888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569623843303262 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569623171826146 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144212222 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576474313593344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8268541111778 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561354647753728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚9899 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187048404 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576474313593344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚813 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331280664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8299754755072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187068442 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144212220 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187048404 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144212222 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629452303930880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584774086351360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7738.752932964 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8299098392576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331293094 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚629513978608640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚7739.440012142 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576530666632192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561409551884288 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187066858 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584831258831360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187086922 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47411466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84155086 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144226238 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584830584452608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144226236 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584831258831360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331293094 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8706239311872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8269357546742 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131566550 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576530666632192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚814 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8299917820416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131566978 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8706077798400 trainer/global_step‚9899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884126364519558e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84155084 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331293094 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚629513978608640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚7739.440012142 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576530666632192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 "Ó8 oD>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561409551884288 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187066858 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584831258831360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187086922 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47411466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84155086 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144226238 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584830584452608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144226236 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584831258831360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331293094 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8706239311872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8269357546742 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131566550 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576530666632192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚814 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8299917820416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131566978 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8706077798400 trainer/global_step‚9899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884126364519558e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84155084 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144913022 # _timestamp‚1.78841267233385e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187974456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚9949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132190974 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579291965535744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187974456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132190523 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47635207 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641283038996992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8340084198400 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572409885170670 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187994494 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8748198182912 _runtime‚7775.321453749 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579291965535744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8340740560896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚564099854281728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8310030888942 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚587632724099584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587632049734144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579291983538688 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144913022 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332907516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144913022 # _timestamp‚1.78841267233385e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187974456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚9949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132190974 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579291965535744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187974456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132190523 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47635207 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641283038996992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8340084198400 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572409885170670 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187994494 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8748198182912 _runtime‚7775.321453749 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579291965535744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8340740560896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚564099854281728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8310030888942 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚587632724099584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587632049734144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579291983538688 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144913022 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332907516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDevi –Rð¸ceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8310192886150 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332919946 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188012974 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚564154758412288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚579348336577536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚587689896579584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187992910 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8341560002048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47640187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144927036 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8310847324114 _runtime‚7776.012225577 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144927038 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚641345746273792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632596515491840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8340903626240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84563486 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641345569208832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8749230781952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚818 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132203224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚564154741372928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9949 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332940012 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84563484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144927036 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚572464934259078 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚587689222200832 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587689896579584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚579348336577536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚579348318574592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8749077657088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚572465605736402 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188012974 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632596491551744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579348318574592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884126730242836e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940012 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332919946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47639740 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0êŠ(üö&™%ðM OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144927036 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8310847324114 _runtime‚7776.012225577 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144927038 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚641345746273792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632596515491840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8340903626240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84563486 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641345569208832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8749230781952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚818 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132203224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚564154741372928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9949 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332940012 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84563484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144927036 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚572464934259078 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚587689222200832 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587689896579584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚579348336577536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚579348318574592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8749077657088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚572465605736402 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188012974 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632596491551744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579348318574592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884126730242836e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940012 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332919946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47639740 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8341560002048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8310192886150 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332919946 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188012974 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚564154758412288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚579348336577536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚587689896579584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187992910 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8341560002048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47640187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414²÷én2š%- _wandbruntime‚7776  _runtime‚7776(¸k [›%Õ 2loss/SRqqt1/assignment_loss‚1.7714279774447967 2loss/SRqqt3/assignment_loss‚0.7946701826407544 /loss/FRt1/detection_loss‚2.7154085364030753 1loss/SRqqt4/detection_loss‚1.7825283195291246 _step‚819 /loss/FRt4/assignment_loss‚2.576349757155594 /loss/FRt3/detection_loss‚1.3870320574325674 /loss/FBt4/detection_loss‚2.3127548032336764 %loss/total_loss‚41.79276656665534 epoch‚8 trainer/global_step‚9949 /loss/FRt2/detection_loss‚2.0782034312963202 0loss/SRqqt2/detection_loss‚1.859095860272646 /loss/FBt3/detection_loss‚0.6027089390080682 0loss/FBt2/assignment_loss‚0.2713775094598532 _runtime‚7776.013348395 0loss/FRt3/assignment_loss‚1.7585296572423448 2loss/SRqqt2/assignment_loss‚1.5574234581989004 2loss/SRqqt4/assignment_loss‚1.2484711083127376 0loss/FBt3/assignment_loss‚0.1055457796224585 /loss/FBt1/detection_loss‚2.7023830711841583 /loss/FBt2/detection_loss‚3.4738263487815857 % _timestamp‚1.7884126730259674e+09 /loss/FRt4/detection_loss‚2.3004148608567765 1loss/SRqqt3/detection_loss‚0.9604750486511376 0loss/FRt1/assignment_loss‚3.3662828777147378 /loss/FRt2/assignment_loss‚3.065182483444611 1loss/FBt1/assignment_loss‚0.24482685858092737 /loss/FBt4/assignment_loss‚0.186748267209623 0loss/SRqqt1/detection_loss‚2.671099372978868S - `œ%Ú 2loss/SRqqt3/assignment_loss‚0.7946701826407544 /loss/FRt1/detection_loss‚2.7154085364030753 1loss/SRqqt4/detection_loss‚1.7825283195291246 _step‚819 /loss/FRt4/assignment_loss‚2.576349757155594 /loss/FRt3/detection_loss‚1.3870320574325674 /loss/FBt4/detection_loss‚2.3127548032336764 %loss/total_loss‚41.79276656665534 epoch‚8 trainer/global_step‚9949 /loss/FRt2/detection_loss‚2.0782034312963202 0loss/SRqqt2/detection_loss‚1.859095860272646 /loss/FBt3/detection_loss‚0.6027089390080682 0loss/FBt2/assignment_loss‚0.2713775094598532 _runtime‚7776.013348395 0loss/FRt3/assignment_loss‚1.7585296572423448 2loss/SRqqt2/assignment_loss‚1.5574234581989004 2loss/SRqqt4/assignment_loss‚1.2484711083127376 0loss/FBt3/assignment_loss‚0.1055457796224585 /loss/FBt1/detection_loss‚2.7023830711841583 /loss/FBt2/detection_loss‚3.4738263487815857 % _timestamp‚1.7884126730259674e+09 /loss/FRt4/detection_loss‚2.3004148608567765 1loss/SRqqt3/detection_loss‚0.9604750486511376 0loss/FRt1/assignment_loss‚3.3662828777147378 /loss/FRt2/assignment_loss‚3.065182483444611 1loss/FBt1/assignment_loss‚0.24482685858092737 /loss/FBt4/assignment_loss‚0.186748267209623 0loss/SRqqt1/detection_loss‚2.671099372978868 2loss/SRqqt1/assignment_loss‚1.7714279774447967³¯™›ß’%‚(:‡ Š†äÔ¾¬† gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.161' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6322 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍzAw€ž%‚(:õ Š†äÔ½‚ý  disk.sda3.out‚ 358.86328125 cpu‚25.356109916671866 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 358.86328125 disk.drbd2266.out‚0 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2810.9296875% disk.nvme0n1.out‚52073.2998046875 disk.dm-0.in‚ 1187.62109375" disk.dm-0.out‚52073.2998046875 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda.out‚ 358.86328125 disk.drbd1716.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375$ memory_percent‚83.22527408599854* proc.memory.percent‚17.156553268432617 disk.sda2.in‚0 disk.rbd1.out‚ 34.58984375 disk.drbd2266.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 2748.37109375 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 2.867452e+06* disk./.usagePercent‚16.578001358601263% disk./.usageGB‚247.02420043945312' disk.nvme0c0n1.out‚52073.2998046875 disk.drbd2080.in‚0 disk.rbd0.in‚0 network.sent‚ 2.4238656e+07# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.sda.in‚ 0.19531253éèö‘Ÿ%‚(:† ™†äÔ®¤ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.655' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.731 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'²oª %‚(:ô ™†äÔ¶©“  network.sent‚ 2.4347606e+07 disk.drbd1716.in‚0 disk.rbd1.out‚ 34.67578125% disk.nvme0n1.out‚52127.8037109375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0) proc.memory.availableMB‚ 2748.37109375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 359.48828125 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.in‚0 disk.dm-0.in‚ 1187.62109375" disk.dm-0.out‚52127.8037109375 disk.drbd2266.out‚0 network.recv‚ 2.874254e+06 disk.sda.out‚ 359.48828125* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 359.48828125* disk./.usagePercent‚16.580017929948472 disk.sda2.out‚0' disk.nvme0c0n1.out‚52127.8037109375! disk.nvme0n1.in‚ 1187.62109375% disk./.usageGB‚247.05424880981445 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0$ memory_percent‚83.22527408599854" proc.memory.rssMB‚ 2810.9296875 cpu‚25.35552196688359´A§ÿ2¡%- _wandbruntime‚7811  _runtime‚7811—ï8îÅ¢%¿ trainer/global_step‚9999 % _timestamp‚1.7884127088569965e+09 _runtime‚7811.844450783 _step‚820 % lr-AdamW/pg1‚2.43111204465008e-05 % lr-AdamW/pg2‚2.43111204465008e-05*FÊ£%Ä % _timestamp‚1.7884127088569965e+09 _runtime‚7811.844450783 _step‚820 % lr-AdamW/pg1‚2.43111204465008e-05 % lr-AdamW/pg2‚2.43111204465008e-05 trainer/global_step‚9999´(Ø 2¤%- _wandbruntime‚7811  _runtime‚7811¬óƒ½ ¥%áO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188920546 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8381726366720 B &*batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84963718 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚582109635481088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590490687482368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635617401637376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188900508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚590491361847808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8791368240128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884127088577816e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84963716 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334534368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚9999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334514328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚582109635481088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8381070004224 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8381726366720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644408769877504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575195927038078 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145613822 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590490687482368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575195255560962 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8350866228350 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚566845043770368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145613820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132827898 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145613822 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚582109617478144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188920546 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132828349 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145613820 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8350211790594 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚821 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590490687482368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635617401637376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188900508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚590491361847808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8791368240128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884127088577816e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84963716 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334534368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚9999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334514328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚582109635481088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8381070004224 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8381726366720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644408769877504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575195927038078 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145613822 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590490687482368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575195255560962 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8350866228350 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚566845043770368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145613820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132827898 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145613822 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚582109617478144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188920546 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132828349 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145613820 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8350211790594 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚821 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145627838 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145627836 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188939026 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566899947900928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84971886 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566899964940288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚582165988519936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566864 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188918962 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334546798 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8382545807872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644471099702784 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132840464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8792226489856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84971884 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582165970516992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚635679028434944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582165988519936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188939026 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188918962 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635679052375040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ÈPQ¤©%ðM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188939026 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188918962 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635679052375040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145627838 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145627836 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188939026 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566899947900928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84971886 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566899964940288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚582165988519936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566864 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188918962 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334546798 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8382545807872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644471099702784 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132840464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8792226489856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84971884 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582165970516992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚635679028434944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582165988519936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2¶®Gy"2ª%- _wandbruntime‚7812  _runtime‚7812?ˆ [«%Õ 0loss/FRt3/assignment_loss‚1.4561761195460956 2loss/SRqqt2/assignment_loss‚1.3858477715402842 1loss/SRqqt3/assignment_loss‚0.988026044346274 /loss/FRt4/detection_loss‚2.1651519185529566 0loss/SRqqt1/detection_loss‚2.457184153795242 1loss/SRqqt2/detection_loss‚1.8365883040039435 /loss/FRt1/detection_loss‚2.6879080943763256 /loss/FBt3/detection_loss‚0.6391502803773619 1loss/FBt2/assignment_loss‚0.24261573574040085 /loss/FRt3/detection_loss‚1.4266186604121078 _runtime‚ 7812.53552876 /loss/FBt1/detection_loss‚2.1759873231252036 _step‚823 % _timestamp‚1.7884127095480917e+09 /loss/FRt2/assignment_loss‚3.140938072762591 2loss/SRqqt1/assignment_loss‚1.1644491985440255 2loss/SRqqt4/assignment_loss‚1.4418041140102504 1loss/FBt3/assignment_loss‚0.11514822848274194 1loss/SRqqt3/detection_loss‚1.0012945146078156 /loss/FBt4/detection_loss‚2.6641388177871703 epoch‚8 &loss/total_loss‚38.996938860076185 0loss/FBt1/assignment_loss‚0.2715709310238405 0loss/FBt4/assignment_loss‚0.0717314982553944 .loss/FRt2/detection_loss‚2.025084113103425 /loss/FBt2/detection_loss‚2.1744688153266907 trainer/global_step‚9999 /loss/FRt1/assignment_loss‚2.798232893645763 /loss/FRt4/assignment_loss‚2.892745170953139 1loss/SRqqt4/detection_loss‚1.7740780857571385Þ¦´`¬%Ú 2loss/SRqqt2/assignment_loss‚1.3858477715402842 1loss/SRqqt3/assignment_loss‚0.988026044346274 /loss/FRt4/detection_loss‚2.1651519185529566 0loss/SRqqt1/detection_loss‚2.457184153795242 1loss/SRqqt2/detection_loss‚1.8365883040039435 /loss/FRt1/detection_loss‚2.6879080943763256 /loss/FBt3/detection_loss‚0.6391502803773619 1loss/FBt2/assignment_loss‚0.24261573574040085 /loss/FRt3/detection_loss‚1.4266186604121078 _runtime‚ 7812.53552876 /loss/FBt1/detection_loss‚2.1759873231252036 _step‚823 % _timestamp‚1.7884127095480917e+09 /loss/FRt2/assignment_loss‚3.140938072762591 2loss/SRqqt1/assignment_loss‚1.1644491985440255 2loss/SRqqt4/assignment_loss‚1.4418041140102504 1loss/FBt3/assignment_loss‚0.11514822848274194 1loss/SRqqt3/detection_loss‚1.0012945146078156 /loss/FBt4/detection_loss‚2.6641388177871703 epoch‚8 &loss/total_loss‚38.996938860076185 0loss/FBt1/assignment_loss‚0.2715709310238405 0loss/FBt4/assignment_loss‚0.0717314982553944 .loss/FRt2/detection_loss‚2.025084113103425 /loss/FBt2/detection_loss‚2.1744688153266907 trainer/global_step‚9999 /loss/FRt1/assignment_loss‚2.798232893645763 /loss/FRt4/assignment_loss‚2.892745170953139 1loss/SRqqt4/detection_loss‚1.7740780857571385 0loss/FRt3/assignment_loss‚1.4561761195460956·›”¸’­%‚(:‡ ¨†äÔ¶Ìû gpu.0.gpu‚26.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü!öuy®%‚(:î ¨†äÔ¤®… $ memory_percent‚82.69057273864746% disk./.usageGB‚247.05372619628906 disk.drbd2266.out‚0! disk.dm-0.out‚52209.642578125 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.sda1.out‚0* disk./.usagePercent‚16.579982856916285 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.sda.in‚ 0.1953125 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 360.1484375 disk.drbd2265.in‚0( proc.memory.availableMB‚ 2835.9765625 network.recv‚ 2.878142e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375$ disk.nvme0n1.out‚52209.642578125* proc.memory.percent‚17.156553268432617# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚ 34.67578125" proc.memory.rssMB‚ 2810.9296875 network.sent‚ 2.4352999e+07 disk.drbd2266.in‚0 disk.dm-0.in‚ 1187.62109375 disk.sda.out‚ 360.1484375 cpu‚25.354912928151503& disk.nvme0c0n1.out‚52209.642578125 disk.rbd1.in‚0 disk.sda3.out‚ 360.1484375×X¯%‚(:’ ·†äÔŸâ’ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.319' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.263799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÀœz°%‚(:ï ·†äԱȢ  disk.drbd1716.out‚0# disk.dm-0.out‚52311.30224609375 disk.drbd2265.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 cpu‚25.3543224757335 disk.sda3.out‚ 360.5546875 disk.rbd0.out‚0" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.rbd0.in‚0( disk./.usagePercent‚16.5818417276224# disk.nvme0c0n1.in‚ 1187.62109375& disk.nvme0n1.out‚52311.30224609375 disk.dm-1.out‚ 360.5546875 disk.sda2.in‚0 network.sent‚ 2.4461708e+07% disk./.usageGB‚247.08142471313477 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚52311.30224609375 disk.sda1.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 2876.6640625* proc.memory.percent‚17.156553268432617 network.recv‚ 2.88509e+06 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125$ memory_percent‚82.44223594665527 disk.drbd2080.in‚0 disk.rbd1.out‚ 34.73828125! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.sda.out‚ 360.5546875 disk.sda1.in‚0˜D±¢’±%‚(:‡ ƆäÔΕ€ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¨ƒ„v²%‚(:ë ƆäÔÛÒý  disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0$ disk./.usageGB‚247.0965576171875 disk.dm-1.out‚ 361.26953125 disk.sda.out‚ 361.26953125! disk.nvme0n1.in‚ 1187.62109375" disk.nvme0n1.out‚ 52370.3515625) proc.memory.availableMB‚ 2876.41796875 disk.dm-0.out‚ 52370.3515625 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 34.73828125$ memory_percent‚82.44373798370361 disk.drbd1716.out‚0$ disk.nvme0c0n1.out‚ 52370.3515625 network.recv‚ 2.889469e+06 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.sda1.out‚0 network.sent‚ 2.4467602e+07 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375* disk./.usagePercent‚16.582857309510732 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2810.9296875* proc.memory.percent‚17.156553268432617 cpu‚25.353744380648926 proc.cpu.threads‚17 disk.sda3.out‚ 361.26953125 disk.sda2.out‚0 disk.drbd1716.in‚0žãíÓ2³%- _wandbruntime‚7848  _runtime‚7848|O‘Ç´%Á & lr-AdamW/pg2‚2.319583148090213e-05 trainer/global_step‚10049 $ _timestamp‚1.788412745502496e+09 _runtime‚7848.489959021 _step‚824 & lr-AdamW/pg1‚2.319583148090213e-05……ð̵%Æ _runtime‚7848.489959021 _step‚824 & lr-AdamW/pg1‚2.319583148090213e-05 & lr-AdamW/pg2‚2.319583148090213e-05 trainer/global_step‚10049 $ _timestamp‚1.788412745502496e+09¸zF 2¶%- _wandbruntime‚7848  _runtime‚7848 t+è'·%âO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚647534196138496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189846598 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189826560 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚593349999596032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647534014865920 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8422055810048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8834257617920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336141180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336161220 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584927269420544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146314620 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638699914580480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577981297428370 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚593349999596032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8391701567758 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584927287423488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚10049 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133464311 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569590250298368 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚593349325230592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584927269420544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577981968905486 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8391047130002 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146314622 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189846598 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚593349325230592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48092195 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884127455031743e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146314620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85372116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48092655 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8834100285440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146314622 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8422712172544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336141180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336161220 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584927269420544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146314620 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638699914580480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577981297428370 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚593349999596032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8391701567758 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584927287423488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚10049 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133464311 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569590250298368 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚593349325230592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584927269420544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577981968905486 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8391047130002 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146314622 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189846598 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚593349325230592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48092195 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884127455031743e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146314620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85372116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48092655 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8834100285440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146314622 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8422712172544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚593349999596032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647534014865920 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8422055810048¹7Ý«¥2¹%- _wandbruntime‚7849  _runtime‚78490µh$ò&º%ìM BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48096695 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8392518002930 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189865078 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146328638 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8834942755328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336173650 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7849.167076311 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146328636 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584983622459392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336173650 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189845014 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146328638 trainer/global_step‚10049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚593407172076032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85380284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336193716 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133476979 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8391863564966 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8423531613696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584983640462336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8422875237888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146328636 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8423531613696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189865078 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8835114754560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚593407172076032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578037017993894 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638761589258240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85380286 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48097156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584983640462336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584983622459392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚647596508073472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414Äé'Æ÷&»%ñM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48096695 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8392518002930 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189865078 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146328638 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8834942755328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336173650 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7849.167076311 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146328636 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584983622459392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336173650 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189845014 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146328638 trainer/global_step‚10049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚593407172076032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85380284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336193716 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133476979 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8391863564966 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8423531613696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584983640462336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8422875237888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146328636 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8423531613696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189865078 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8835114754560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚593407172076032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578037017993894 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638761589258240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85380286 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48097156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584983640462336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584983622459392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚647596508073472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0º«Dz2¼%- _wandbruntime‚7849  _runtime‚7849ððSV½%Ð /loss/FRt3/detection_loss‚1.5541327876203201 1loss/SRqqt2/detection_loss‚1.8077936324736346 2loss/SRqqt3/assignment_loss‚0.8649741581175476 .loss/FBt1/detection_loss‚2.265921974182129 epoch‚8 trainer/global_step‚10049 _runtime‚ 7849.16841137 0loss/SRqqt1/detection_loss‚2.174077659118466 /loss/FRt2/assignment_loss‚3.120581083354496 1loss/SRqqt4/assignment_loss‚1.420286897439183 1loss/SRqqt3/detection_loss‚1.0568706645281054 .loss/FBt2/detection_loss‚2.131142516930898 /loss/FBt3/detection_loss‚0.4508965497587396 % _timestamp‚1.7884127461807504e+09 0loss/FRt1/assignment_loss‚2.9820857206980387 0loss/FBt4/assignment_loss‚0.5337991030479315 0loss/SRqqt4/detection_loss‚1.695519427981293 0loss/FRt4/assignment_loss‚2.6829302847385406 %loss/total_loss‚39.69060975246644 1loss/FBt3/assignment_loss‚0.18604491896516004 .loss/FRt4/detection_loss‚2.186567883491516 .loss/FBt4/detection_loss‚1.915327352285385 0loss/FBt2/assignment_loss‚0.3987538699887712 /loss/FRt1/detection_loss‚2.9091896414756775 _step‚827 0loss/FRt3/assignment_loss‚1.6858446434432386 2loss/SRqqt1/assignment_loss‚1.6422095684016622 1loss/SRqqt2/assignment_loss‚1.353397029330549 0loss/FBt1/assignment_loss‚0.5276631377171725 .loss/FRt2/detection_loss‚2.144599247377989ÓàæX[¾%Õ 1loss/SRqqt4/assignment_loss‚1.420286897439183 1loss/SRqqt3/detection_loss‚1.0568706645281054 .loss/FBt2/detection_loss‚2.131142516930898 /loss/FBt3/detection_loss‚0.4508965497587396 % _timestamp‚1.7884127461807504e+09 0loss/FRt1/assignment_loss‚2.9820857206980387 0loss/FBt4/assignment_loss‚0.5337991030479315 0loss/SRqqt4/detection_loss‚1.695519427981293 0loss/FRt4/assignment_loss‚2.6829302847385406 %loss/total_loss‚39.69060975246644 1loss/FBt3/assignment_loss‚0.18604491896516004 .loss/FRt4/detection_loss‚2.186567883491516 .loss/FBt4/detection_loss‚1.915327352285385 0loss/FBt2/assignment_loss‚0.3987538699887712 /loss/FRt1/detection_loss‚2.9091896414756775 _step‚827 0loss/FRt3/assignment_loss‚1.6858446434432386 2loss/SRqqt1/assignment_loss‚1.6422095684016622 1loss/SRqqt2/assignment_loss‚1.353397029330549 0loss/FBt1/assignment_loss‚0.5276631377171725 .loss/FRt2/detection_loss‚2.144599247377989 /loss/FRt3/detection_loss‚1.5541327876203201 1loss/SRqqt2/detection_loss‚1.8077936324736346 2loss/SRqqt3/assignment_loss‚0.8649741581175476 .loss/FBt1/detection_loss‚2.265921974182129 epoch‚8 trainer/global_step‚10049 _runtime‚ 7849.16841137 0loss/SRqqt1/detection_loss‚2.174077659118466 /loss/FRt2/assignment_loss‚3.120581083354496»¸Ð\G¿%‚(:’ Õ†äÔªˆ´ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.296' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.459199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûãþ°zÀ%‚(:ï Õ†äÔã® # disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2080.out‚0* proc.memory.percent‚17.156553268432617 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 361.87890625 disk.sda.in‚ 0.1953125) proc.memory.availableMB‚ 2876.38671875 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚52428.05078125 disk.sda.out‚ 361.87890625% disk./.usageGB‚247.12126541137695 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 network.recv‚ 2.89409e+06 disk.sda1.in‚0 disk.dm-0.in‚ 1187.62109375 cpu‚25.353170661737227 disk.sda1.out‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 361.87890625 disk.drbd2266.in‚0# memory_percent‚82.4439287185669 network.sent‚ 2.4473921e+07 disk.rbd1.out‚ 34.7705078125# disk.nvme0n1.out‚52428.05078125 disk.drbd2265.in‚0* disk./.usagePercent‚16.584515470310006 disk.sda2.in‚0 disk.dm-0.out‚52428.05078125 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0Ù  Á%‚(:’ ä†äÔᔓ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.889' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.777800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B–#6Â%‚(:ô ä†äÔ€ýä  disk.rbd1.in‚0" disk.dm-0.out‚52533.3271484375 disk.dm-1.out‚ 362.55859375 disk.sda1.out‚0* proc.memory.percent‚17.156553268432617) disk./.usagePercent‚16.58733948942021# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚ 34.7705078125 disk.sda.out‚ 362.55859375$ memory_percent‚82.44242668151855 network.sent‚ 2.4582136e+07 disk.dm-0.in‚ 1187.62109375( proc.memory.availableMB‚ 2876.6328125 network.recv‚ 2.900214e+06 disk.drbd1716.out‚0 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 362.55859375 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1187.62109375% disk.nvme0n1.out‚52533.3271484375 disk.drbd1716.in‚0 disk.sda3.in‚ 0.1953125 disk.sda.in‚ 0.1953125 cpu‚25.35255388383182% disk./.usageGB‚247.16334533691406' disk.nvme0c0n1.out‚52533.3271484375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0nÉï2Ã%- _wandbruntime‚7884  _runtime‚7884åÅ <ÊÄ%Ä ' lr-AdamW/pg2‚2.2104369485646367e-05 trainer/global_step‚10099 % _timestamp‚1.7884127816990545e+09 _runtime‚7884.686455535 _step‚828 ' lr-AdamW/pg1‚2.2104369485646367e-05î›âÏÅ%É trainer/global_step‚10099 % _timestamp‚1.7884127816990545e+09 _runtime‚7884.686455535 _step‚828 ' lr-AdamW/pg1‚2.2104369485646367e-05 ' lr-AdamW/pg2‚2.2104369485646367e-05¼òP¥02Æ%- _wandbruntime‚7884  _runtime‚7884C[ ½Ç%áO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190752612 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48323580 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190772650 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596207962978816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147015422 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8877461426688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337788072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚587744921362944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8463041615872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134104571 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚580768010772894 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚587744939365888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190772650 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190752612 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8463697978368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641782475403776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚596208637344256 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147015420 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 $ _timestamp‚1.788412781699765e+09 _runtime‚7884.687412892 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8463697978368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚596208637344256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48324053 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚10099 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85780516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚587744939365888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚650659749266432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147015420 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8463041615872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134104571 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚580768010772894 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚587744939365888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190772650 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190752612 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8463697978368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641782475403776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚596208637344256 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147015420 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 $ _timestamp‚1.788412781699765e+09 _runtime‚7884.687412892 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8463697978368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚596208637344256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48324053 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚10099 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85780516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚587744939365888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚650659749266432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147015420 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8463041615872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596207962978816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147015422 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8877461426688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337788072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1½Ç÷H™2É%- _wandbruntime‚7885  _runtime‚7885MÿÂò&Ê%ìM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337800502 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134116933 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85788686 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650722461765632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650722282603520 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337800502 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641844126141440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8878335624192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚572390377996288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8433353342338 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8463861043712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190791130 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884127823807871e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8463861043712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85788684 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147029438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚596265809824256 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚587801292404736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820568 _step‚830 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8464517419520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚587801274401792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48328249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134117398 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337820568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8878180402176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147029436 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚580823731338626 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596265135445504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8464517419520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚587801274401792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48328712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8432698904374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190791130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641844102201344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844ªAÁm£Ë%ñM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85788686 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650722461765632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650722282603520 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337800502 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641844126141440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8878335624192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚572390377996288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8433353342338 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8463861043712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190791130 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884127823807871e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8463861043712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85788684 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147029438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚596265809824256 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚587801292404736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820568 _step‚830 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8464517419520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚587801274401792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48328249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134117398 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337820568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8878180402176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147029436 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚580823731338626 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596265135445504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8464517419520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚587801274401792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48328712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8432698904374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190791130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641844102201344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337800502 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134116933 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928¾[nãF2Ì%- _wandbruntime‚7885  _runtime‚7885{žšm^Í%Ø /loss/FRt2/assignment_loss‚2.574615044253213 2loss/SRqqt4/assignment_loss‚1.5319313588884793 trainer/global_step‚10099 2loss/SRqqt2/assignment_loss‚0.9770904437838388 1loss/FBt3/assignment_loss‚0.10589591916291961 3loss/FBt4/assignment_loss‚0.0013863915518470549 1loss/SRqqt1/assignment_loss‚1.514449899715762 2loss/SRqqt3/assignment_loss‚0.9626723364668174 0loss/SRqqt1/detection_loss‚2.437663999296004 1loss/SRqqt4/detection_loss‚1.8819225044274817 /loss/FBt1/detection_loss‚2.2597572671042547 /loss/FBt4/detection_loss‚2.8912638127803802 1loss/FBt1/assignment_loss‚0.10150936701231533 0loss/SRqqt2/detection_loss‚1.610114669826414 epoch‚8 0loss/FRt3/assignment_loss‚1.0624919078566812 .loss/FRt1/detection_loss‚3.131483351190885 /loss/FBt2/detection_loss‚2.8176027983427048 $ _timestamp‚1.788412782382485e+09 _runtime‚7885.369824183 /loss/FRt4/assignment_loss‚2.975282535238086 /loss/FRt2/detection_loss‚2.1990637332321006 /loss/FRt4/detection_loss‚2.2173427632957137 1loss/SRqqt3/detection_loss‚0.9037759401835501 /loss/FBt3/detection_loss‚0.7027463411366065 %loss/total_loss‚39.46151709647373 /loss/FRt1/assignment_loss‚3.002544015645981 1loss/FBt2/assignment_loss‚0.07780423871008235 /loss/FRt3/detection_loss‚1.5211064573716033 _step‚831O6ÝcÎ%Ý _step‚831 /loss/FRt2/assignment_loss‚2.574615044253213 2loss/SRqqt4/assignment_loss‚1.5319313588884793 trainer/global_step‚10099 2loss/SRqqt2/assignment_loss‚0.9770904437838388 1loss/FBt3/assignment_loss‚0.10589591916291961 3loss/FBt4/assignment_loss‚0.0013863915518470549 1loss/SRqqt1/assignment_loss‚1.514449899715762 2loss/SRqqt3/assignment_loss‚0.9626723364668174 0loss/SRqqt1/detection_loss‚2.437663999296004 1loss/SRqqt4/detection_loss‚1.8819225044274817 /loss/FBt1/detection_loss‚2.2597572671042547 /loss/FBt4/detection_loss‚2.8912638127803802 1loss/FBt1/assignment_loss‚0.10150936701231533 0loss/SRqqt2/detection_loss‚1.610114669826414 epoch‚8 0loss/FRt3/assignment_loss‚1.0624919078566812 .loss/FRt1/detection_loss‚3.131483351190885 /loss/FBt2/detection_loss‚2.8176027983427048 $ _timestamp‚1.788412782382485e+09 _runtime‚7885.369824183 /loss/FRt4/assignment_loss‚2.975282535238086 /loss/FRt2/detection_loss‚2.1990637332321006 /loss/FRt4/detection_loss‚2.2173427632957137 1loss/SRqqt3/detection_loss‚0.9037759401835501 /loss/FBt3/detection_loss‚0.7027463411366065 %loss/total_loss‚39.46151709647373 /loss/FRt1/assignment_loss‚3.002544015645981 1loss/FBt2/assignment_loss‚0.07780423871008235 /loss/FRt3/detection_loss‚1.5211064573716033¿ê½Ï%‚(:… ó†äÔ•Þ“ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ÓµjÐ%‚(:ß ó†äÔ«’  disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 2876.5703125* disk./.usagePercent‚16.590038320846173 disk.rbd1.out‚34.83349609375 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda3.out‚363.125# disk.nvme0c0n1.in‚ 1187.62109375% disk.nvme0c0n1.out‚52595.66796875* proc.memory.percent‚17.156553268432617# disk.nvme0n1.out‚52595.66796875 disk.sda.out‚363.125" proc.memory.rssMB‚ 2810.9296875 disk.drbd2266.out‚0 disk.rbd0.in‚0 network.recv‚ 2.904327e+06% disk./.usageGB‚247.20355987548828! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0 cpu‚25.3519914260051 disk.dm-0.out‚52595.66796875 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚363.125 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0$ memory_percent‚82.44280815124512 proc.cpu.threads‚17 network.sent‚ 2.458821e+07 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0évm„’Ñ%‚(:‡ ‚‡äÔàñü gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.381' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€/¶tÒ%‚(:é ‚‡äÔ¥çû disk.drbd2266.in‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚52659.33984375 disk.drbd1716.in‚0! disk.nvme0n1.in‚ 1187.62109375# disk.nvme0n1.out‚52659.33984375 disk.dm-0.out‚52659.33984375 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.rbd1.out‚34.83349609375 disk.dm-1.out‚ 363.765625 disk.drbd2265.out‚0 disk.sda3.out‚ 363.765625 disk.drbd2080.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.156553268432617 network.recv‚ 2.911285e+06# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.in‚0 disk.drbd2266.out‚0% disk./.usageGB‚247.23456573486328 disk.sda.in‚ 0.1953125 disk.sda.out‚ 363.765625) disk./.usagePercent‚16.59211915008448 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125$ memory_percent‚82.44280815124512 disk.sda2.out‚0 disk.dm-0.in‚ 1187.62109375 network.sent‚ 2.4696879e+07 disk.dm-1.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 2876.5703125 cpu‚25.35138983585988¹«p-Ó%‚(:’ ‘‡äÔ‚ß° gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚155.758' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.151600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ìw„Ô%‚(:ù ‘‡äÔóÜ  disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375 cpu‚25.350788428423325 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚52729.28466796875 disk.drbd2266.in‚0$ memory_percent‚82.44297504425049 disk.rbd1.in‚0 disk.dm-0.in‚ 1187.62109375* proc.memory.percent‚17.156553268432617 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 34.896484375 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 364.07421875) proc.memory.availableMB‚ 2876.54296875 disk.drbd2265.out‚0 network.recv‚ 2.916183e+06% disk./.usageGB‚247.26555252075195 disk.drbd2080.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚52729.28466796875 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda.in‚ 0.1953125 disk.sda.out‚ 364.07421875# disk.dm-0.out‚52729.28466796875 disk.dm-1.out‚ 364.07421875" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 2.4703234e+07* disk./.usagePercent‚16.594198699285112 disk.drbd2080.out‚01P”2Õ%- _wandbruntime‚7921  _runtime‚7921ÊÙÈÖ% & lr-AdamW/pg1‚2.103696191977745e-05 & lr-AdamW/pg2‚2.103696191977745e-05 trainer/global_step‚10149 % _timestamp‚1.7884128180713758e+09 _runtime‚7921.058906664 _step‚832µb÷øÍ×%Ç & lr-AdamW/pg1‚2.103696191977745e-05 & lr-AdamW/pg2‚2.103696191977745e-05 trainer/global_step‚10149 % _timestamp‚1.7884128180713758e+09 _runtime‚7921.058906664 _step‚832ÀŸÅê2Ø%- _wandbruntime‚7921  _runtime‚7921ìSè'Ù%âO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653785645551616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8473372246574 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚590562573305344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147716222 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8472717808818 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚590562573305344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644865012286976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8920461252096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48553701 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575080663354368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599066600727040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653785449598976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86188918 _step‚833 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7921.059709098 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚599067275092480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚590562591308288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8504027421696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191678664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8504027421696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599066600727040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653785449598976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86188918 _step‚833 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7921.059709098 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚599067275092480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚590562591308288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8504027421696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191678664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8504027421696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8472717808818 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚590562573305344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644865012286976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8920461252096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48553701 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575080663354368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134755264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48558180 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8921325861888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚575135567484928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8504846849536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚599123773193728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339427354 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8474188681746 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8505503225344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.curre/œåØnt‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147730238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644926663024640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447420 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191717182 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚590618944347136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86197086 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653847964946432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191697118 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590618926344192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚583609773206034 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8505503225344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚599124447572480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚583609101728710 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191697118 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚599124447572480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚590618926344192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339447420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚10149 _runtime‚7921.748516089 _step‚834 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884128187604322e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86197084 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8504846849536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0D¯ér÷&Ý%ñM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884128187604322e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86197084 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8504846849536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134755726 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134755264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48558180 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8921325861888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚575135567484928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8504846849536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚599123773193728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339427354 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8474188681746 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8505503225344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147730238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644926663024640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447420 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191717182 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚590618944347136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86197086 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653847964946432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191697118 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590618926344192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚583609773206034 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8505503225344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚599124447572480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚583609101728710 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191697118 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚599124447572480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚590618926344192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339447420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚10149 _runtime‚7921.748516089 _step‚834Âw+.2Þ%- _wandbruntime‚7921  _runtime‚7921jZXß%Ò /loss/FBt3/detection_loss‚0.5176564733603899 .loss/FBt4/detection_loss‚2.476011887192726 1loss/FBt1/assignment_loss‚0.10982057031449383 0loss/FBt2/assignment_loss‚0.2632177921145085 /loss/FRt1/detection_loss‚2.6614414006471634 .loss/FRt3/detection_loss‚1.537090240448129 _runtime‚7921.749658585 2loss/SRqqt3/assignment_loss‚0.8191051430043739 .loss/FRt2/detection_loss‚2.231674653788408 _step‚835 trainer/global_step‚10149 1loss/SRqqt4/assignment_loss‚1.119185323041736 1loss/FBt3/assignment_loss‚0.08943257560977308 1loss/FBt4/assignment_loss‚0.20152441076780292 /loss/FBt1/detection_loss‚2.9909123049841986 /loss/FBt2/detection_loss‚2.1608783389840807 /loss/FRt2/assignment_loss‚2.712828534344832 0loss/FRt4/assignment_loss‚2.4799756915011306 0loss/FRt3/assignment_loss‚1.0777851635334539 1loss/SRqqt3/detection_loss‚0.8923345443264052 %loss/total_loss‚38.98271070222405 $ _timestamp‚1.788412818762292e+09 2loss/SRqqt1/assignment_loss‚1.7063420198744896 .loss/FRt4/detection_loss‚2.369538172128353 epoch‚8 /loss/FRt1/assignment_loss‚3.225378155708313 0loss/SRqqt1/detection_loss‚2.230449041633895 0loss/SRqqt2/detection_loss‚1.731816975907846 1loss/SRqqt2/assignment_loss‚1.459273921224204 1loss/SRqqt4/detection_loss‚1.9190373677833408øÖü>]à%× 1loss/SRqqt4/detection_loss‚1.9190373677833408 /loss/FBt3/detection_loss‚0.5176564733603899 .loss/FBt4/detection_loss‚2.476011887192726 1loss/FBt1/assignment_loss‚0.10982057031449383 0loss/FBt2/assignment_loss‚0.2632177921145085 /loss/FRt1/detection_loss‚2.6614414006471634 .loss/FRt3/detection_loss‚1.537090240448129 _runtime‚7921.749658585 2loss/SRqqt3/assignment_loss‚0.8191051430043739 .loss/FRt2/detection_loss‚2.231674653788408 _step‚835 trainer/global_step‚10149 1loss/SRqqt4/assignment_loss‚1.119185323041736 1loss/FBt3/assignment_loss‚0.08943257560977308 1loss/FBt4/assignment_loss‚0.20152441076780292 /loss/FBt1/detection_loss‚2.9909123049841986 /loss/FBt2/detection_loss‚2.1608783389840807 /loss/FRt2/assignment_loss‚2.712828534344832 0loss/FRt4/assignment_loss‚2.4799756915011306 0loss/FRt3/assignment_loss‚1.0777851635334539 1loss/SRqqt3/detection_loss‚0.8923345443264052 %loss/total_loss‚38.98271070222405 $ _timestamp‚1.788412818762292e+09 2loss/SRqqt1/assignment_loss‚1.7063420198744896 .loss/FRt4/detection_loss‚2.369538172128353 epoch‚8 /loss/FRt1/assignment_loss‚3.225378155708313 0loss/SRqqt1/detection_loss‚2.230449041633895 0loss/SRqqt2/detection_loss‚1.731816975907846 1loss/SRqqt2/assignment_loss‚1.459273921224204ÃMD€{’á%‚(:‡  ‡äÔò‚‡ gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šZ´Öyâ%‚(:î  ‡äÔŠ¢÷ disk.dm-1.in‚ 0.1953125 disk.dm-0.out‚52808.13671875 disk.sda.out‚ 364.61328125 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 cpu‚25.350222833504883 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17* disk./.usagePercent‚16.596362986979578 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 2876.265625# disk.nvme0n1.out‚52808.13671875 disk.dm-1.out‚ 364.61328125% disk.nvme0c0n1.out‚52808.13671875 disk.rbd1.in‚0 disk.rbd1.out‚ 34.896484375$ memory_percent‚82.44466781616211% disk./.usageGB‚247.29780197143555 network.sent‚ 2.4708658e+07 network.recv‚ 2.920054e+06 disk.sda.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2810.9296875! disk.nvme0n1.in‚ 1187.62109375 disk.sda3.out‚ 364.61328125 disk.rbd0.in‚0 disk.sda2.in‚0^l® ’ã%‚(:‡ ¯‡äԣȡ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`ž}ä%‚(:ò ¯‡äÔ‚¸§  disk.rbd1.in‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 365.05859375 network.recv‚ 2.926713e+06 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2080.in‚0 cpu‚25.3496556969678 proc.cpu.threads‚17 disk.dm-0.in‚ 1187.62109375" disk.dm-0.out‚52885.9130859375 disk.dm-1.out‚ 365.05859375 disk.drbd2080.out‚0 disk.rbd0.out‚0$ memory_percent‚82.44180679321289' proc.memory.availableMB‚ 2876.734375" proc.memory.rssMB‚ 2810.9296875 disk.sda1.out‚0% disk.nvme0n1.out‚52885.9130859375 disk.drbd2266.in‚0* proc.memory.percent‚17.156553268432617 network.sent‚ 2.4817411e+07% disk./.usageGB‚247.32651138305664 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375! disk.nvme0n1.in‚ 1187.62109375* disk./.usagePercent‚16.598289699682272 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 34.896484375 disk.sda1.in‚0' disk.nvme0c0n1.out‚52885.9130859375 disk.sda.in‚ 0.1953125 disk.sda.out‚ 365.05859375¤{­2å%- _wandbruntime‚7957  _runtime‚7957 ÅOÊæ%Ä trainer/global_step‚10199 % _timestamp‚1.7884128546223962e+09 _runtime‚7957.609829444 _step‚836 ' lr-AdamW/pg1‚1.9993831229432135e-05 ' lr-AdamW/pg2‚1.9993831229432135e-05êFØ‹Ïç%É ' lr-AdamW/pg2‚1.9993831229432135e-05 trainer/global_step‚10199 % _timestamp‚1.7884128546223962e+09 _runtime‚7957.609829444 _step‚836 ' lr-AdamW/pg1‚1.9993831229432135e-05Ä&KÓ2è%- _wandbruntime‚7957  _runtime‚7957;ÞÇÑé%âO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚586339423030594 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86597316 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚837 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148417022 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192604716 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8963541138944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8513553148226 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192624754 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚7957.610692312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚593380225247744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8514207585982 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192624754 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚593380243250688 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135380735 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8963700568576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚601925238475264 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚601925912840704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8545669590016 % _timestamp‚1.7884128546230934e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48783419 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656911066369024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚656911249738752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341021736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135381159 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586340094507710 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8545013227520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148417020 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86597318 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚601925238475264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341041776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928AcÈŸí'ê%çO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148417022 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192604716 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8963541138944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8513553148226 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192624754 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚7957.610692312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚593380225247744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8514207585982 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192624754 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚593380243250688 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135380735 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8963700568576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚601925238475264 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚601925912840704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8545669590016 % _timestamp‚1.7884128546230934e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48783419 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656911066369024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚656911249738752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341021736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135381159 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586340094507710 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8545013227520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148417020 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86597318 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚601925238475264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341041776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚586339423030594 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86597316 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚837Åâ•òu2ë%- _wandbruntime‚7958  _runtime‚7958;÷ð&ì%êM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚586395143596118 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577880774012928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7958.287033749 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8546489031168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚586395815073442 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577880791052288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648009199907840 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148431038 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135393475 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚10199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656973581554688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚601982410941952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48787991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8545832655360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚593436596289536 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148431038 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86605484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _step‚838 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8964405586944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8515024021154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86605486 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚656973769105408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48788422 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚593436578286592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192623170 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8545832655360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚601983085320704 # _timestamp‚1.78841285529917e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚648009175967744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135393908 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148431036 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192643234 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8514369583190 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872«5áõ&í%ïM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148431038 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86605484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _step‚838 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8964405586944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8515024021154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86605486 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚656973769105408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48788422 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚593436578286592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192623170 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8545832655360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚601983085320704 # _timestamp‚1.78841285529917e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚648009175967744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135393908 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148431036 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192643234 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8514369583190 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚586395143596118 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577880774012928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7958.287033749 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8546489031168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚586395815073442 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577880791052288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648009199907840 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148431038 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135393475 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚10199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656973581554688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚601982410941952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48787991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8545832655360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚593436596289536Æ~ Yª2î%- _wandbruntime‚7958  _runtime‚7958Cá‚ï%Ø 2loss/SRqqt2/assignment_loss‚1.5004929902925328 1loss/SRqqt3/assignment_loss‚1.002183641130426 trainer/global_step‚10199 _runtime‚7958.288053964 1loss/SRqqt1/assignment_loss‚1.350134096896419 /loss/FRt1/detection_loss‚2.6255085984865825 &loss/total_loss‚38.812696739395946 0loss/FBt2/assignment_loss‚0.3595363389890322 /loss/FRt4/detection_los5B»þÜs‚2.3855534361844715 /loss/FBt1/detection_loss‚3.1811584574835643 /loss/FBt4/detection_loss‚1.9515747845172882 0loss/FRt2/assignment_loss‚3.3033672833442687 0loss/FRt4/assignment_loss‚2.8030217547308314 1loss/SRqqt4/detection_loss‚1.7782916674533715 _step‚839 0loss/FBt4/assignment_loss‚0.3049463452713098 .loss/FRt2/detection_loss‚2.192288227078977 1loss/SRqqt1/detection_loss‚2.0708503012473765 0loss/SRqqt3/detection_loss‚0.835921495767503 epoch‚8 % _timestamp‚1.7884128553007228e+09 /loss/FRt3/detection_loss‚1.6566034110361023 1loss/SRqqt2/detection_loss‚1.9213804247580013 /loss/FBt2/detection_loss‚1.7983371181921526 /loss/FBt3/detection_loss‚0.5865322903456339 .loss/FRt1/assignment_loss‚2.78441959089703 2loss/SRqqt4/assignment_loss‚1.2619468630291522 2loss/FBt1/assignment_loss‚0.011654204040366625 0loss/FRt3/assignment_loss‚1.0873562871364124 1loss/FBt3/assignment_loss‚0.05963713108714748V1ýXcð%Ý .loss/FRt2/detection_loss‚2.192288227078977 1loss/SRqqt1/detection_loss‚2.0708503012473765 0loss/SRqqt3/detection_loss‚0.835921495767503 epoch‚8 % _timestamp‚1.7884128553007228e+09 /loss/FRt3/detection_loss‚1.6566034110361023 1loss/SRqqt2/detection_loss‚1.9213804247580013 /loss/FBt2/detection_loss‚1.7983371181921526 /loss/FBt3/detection_loss‚0.5865322903456339 .loss/FRt1/assignment_loss‚2.78441959089703 2loss/SRqqt4/assignment_loss‚1.2619468630291522 2loss/FBt1/assignment_loss‚0.011654204040366625 0loss/FRt3/assignment_loss‚1.0873562871364124 1loss/FBt3/assignment_loss‚0.05963713108714748 2loss/SRqqt2/assignment_loss‚1.5004929902925328 1loss/SRqqt3/assignment_loss‚1.002183641130426 trainer/global_step‚10199 _runtime‚7958.288053964 1loss/SRqqt1/assignment_loss‚1.350134096896419 /loss/FRt1/detection_loss‚2.6255085984865825 &loss/total_loss‚38.812696739395946 0loss/FBt2/assignment_loss‚0.3595363389890322 /loss/FRt4/detection_loss‚2.3855534361844715 /loss/FBt1/detection_loss‚3.1811584574835643 /loss/FBt4/detection_loss‚1.9515747845172882 0loss/FRt2/assignment_loss‚3.3033672833442687 0loss/FRt4/assignment_loss‚2.8030217547308314 1loss/SRqqt4/detection_loss‚1.7782916674533715 _step‚839 0loss/FBt4/assignment_loss‚0.3049463452713098ÇûöVñ%‚(:’ ¾‡äÔëÖ¯ gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.051' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.410199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í´÷­ò%‚(:ö ¾‡äÔæÊ¿  disk.dm-1.out‚ 365.97265625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 365.97265625 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125* disk./.usagePercent‚16.599913299464426% disk./.usageGB‚247.35070419311523 disk.drbd2265.in‚0 disk.drbd2080.in‚0 cpu‚25.349091814560083 disk.drbd1716.in‚0 disk.sda3.out‚ 365.97265625 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚52939.4833984375" disk.dm-0.out‚52939.4833984375 disk.drbd2080.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 2876.41796875" proc.memory.rssMB‚ 2810.9296875 network.sent‚ 2.4823286e+07 disk.rbd1.out‚ 34.896484375! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2266.out‚0 disk.dm-0.in‚ 1187.62109375$ memory_percent‚82.44373798370361* proc.memory.percent‚17.156553268432617 network.recv‚ 2.930943e+06 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375% disk.nvme0n1.out‚52939.4833984375§ªV’ó%‚(:‡ ͇äÔ‘ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë €[yô%‚(:î ͇äÔù‡  disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2080.out‚0) proc.memory.availableMB‚ 2876.40234375 proc.cpu.threads‚17 network.recv‚ 2.938403e+06& disk.nvme0n1.out‚53013.21337890625 disk.sda3.out‚ 366.46875 network.sent‚ 2.4932549e+07 disk.dm-1.out‚ 366.46875 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2810.9296875( disk.nvme0c0n1.out‚53013.21337890625 disk.sda.out‚ 366.46875$ memory_percent‚82.44383335113525 cpu‚25.34852662254842) disk./.usagePercent‚16.60180929126301 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.sda2.out‚0# disk.dm-0.out‚53013.21337890625 disk.sda1.in‚0* proc.memory.percent‚17.156553268432617% disk./.usageGB‚247.37895584106445 disk.drbd1716.in‚0 disk.rbd1.out‚ 34.896484375hù¦˜2õ%- _wandbruntime‚7994  _runtime‚7994ìgÝqÊö%Ä trainer/global_step‚10249 % _timestamp‚1.7884128913564718e+09 _runtime‚7994.343898102 _step‚840 ' lr-AdamW/pg1‚1.8975194801482864e-05 ' lr-AdamW/pg2‚1.8975194801482864e-05åô@¸Ï÷%É ' lr-AdamW/pg1‚1.8975194801482864e-05 ' lr-AdamW/pg2‚1.8975194801482864e-05 trainer/global_step‚10249 % _timestamp‚1.7884128913564718e+09 _runtime‚7994.343898102 _step‚840Èê êæ2ø%- _wandbruntime‚7994  _runtime‚7994¸hêè'ù%âO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596197877190144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚10249 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604783876223488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342668628 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87005716 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193550806 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596197877190144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580571076410368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49013913 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651030062113280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9006931452928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8585999033344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596197895193088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555042925390 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884128913571477e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8586655395840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8585999033344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193550806 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660037017506304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554388487634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚580571093449728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149117822 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193530768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596197895193088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589125464898002¡D¬úí'ú%çO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8586655395840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8585999033344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193550806 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660037017506304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554388487634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚580571093449728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149117822 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193530768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596197895193088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589125464898002 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596197877190144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚10249 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604783876223488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342668628 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87005716 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193550806 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596197877190144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580571076410368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49013913 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651030062113280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9006931452928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8585999033344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596197895193088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555042925390 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884128913571477e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ɱ)Ö’û%‚(:‡ ܇äÔ…Ÿ£ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=¦Íìpü%‚(:å ܇äÔ¬£   disk.sda3.out‚ 367.28515625# memory_percent‚82.4439287185669 network.sent‚ 2.4937954e+07 disk.sda1.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2080.in‚0 network.recv‚ 2.942148e+06* disk./.usagePercent‚16.603205812362287 disk.sda2.out‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2810.9296875 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.sda.out‚ 367.28515625 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0) proc.memory.availableMB‚ 2876.38671875 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚ 34.896484375* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17" disk.nvme0c0n1.out‚ 53065.46875 disk.drbd1716.in‚0% disk./.usageGB‚247.39976501464844 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.out‚ 367.28515625 disk.drbd2265.out‚0 disk.drbd2266.in‚0 cpu‚25.34794344111281 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.out‚ 53065.46875 disk.dm-0.out‚ 53065.46875 disk.sda3.in‚ 0.1953125¦Ÿ!+2ý%- _wandbruntime‚7995  _runtime‚7995†’Ö»"þ%ëM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149131838 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342681058 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580625980540928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87013886 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193569286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8555859360562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589181185463526 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8586818461184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701124 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚604841723068928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660099524686336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342701124 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8587474836992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8586818461184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596254248231936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604841048690176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193549222 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136032353 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660099341329920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9007787895296 ]KDeviceStatsMonitor.on_train_batchh¾ÔÏ_end/inactive_split_bytes.small_pool.freed‚ 9007628478976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚842 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604841723068928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651091712850944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49018918 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651091736791040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596254248231936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0p…Pö&ÿ%ðM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342681058 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580625980540928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87013886 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193569286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8555859360562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589181185463526 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8586818461184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701124 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚604841723068928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660099524686336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342701124 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8587474836992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8586818461184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596254248231936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604841048690176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193549222 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136032353 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660099341329920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9007787895296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9007628478976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚842 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604841723068928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651091712850944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49018918 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651091736791040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596254248231936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149131838 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ʱGbn2€&- _wandbruntime‚7995  _runtime‚7995à¤);^&Ø /loss/FBt1/detection_loss‚2.1839555452267327 % _timestamp‚1.7884128920534081e+09 2loss/SRqqt1/assignment_loss‚1.7306206102882113 2loss/SRqqt3/assignment_loss‚0.7669822916346178 1loss/FBt2/assignment_loss‚0.36717251657197875 0loss/FBt3/assignment_loss‚0.3676172031295656 1loss/SRqqt1/detection_loss‚2.3460889645985197 /loss/FBt2/detection_loss‚3.7868451277414956 2loss/SRqqt2/assignment_loss‚1.3747328022833576 /loss/FRt1/detection_loss‚2.8847751716772714 0loss/SRqqt3/detection_loss‚0.869011581282724 0loss/FRt1/assignment_loss‚2.5725013290842376 0loss/FRt3/assignment_loss‚1.6589968473303551 1loss/FBt4/assignment_loss‚0.05063441935067203 .loss/FBt4/detection_loss‚4.277999838193257 1loss/FBt1/assignment_loss‚0.22376626854061676 .loss/FRt2/detection_loss‚2.145909079483649 0loss/FRt4/assignment_loss‚3.1868285841412014 2loss/SRqqt4/assignment_loss‚0.9985402896814048 /loss/FRt3/detection_loss‚1.7602444512440878 %loss/total_loss‚43.30767220957044 epoch‚8 /loss/FRt4/detection_loss‚2.3659164167112774 /loss/FBt3/detection_loss‚0.9524323916950919 _runtime‚ 7995.04077384 _step‚843 0loss/FRt2/assignment_loss‚2.7726414483420703 0loss/SRqqt2/detection_loss‚1.875516095960682 1loss/SRqqt4/detection_loss‚1.7879429353773595 trainer/global_step‚10249YÆ’c‚&Ý 0loss/FRt4/assignment_loss‚3.1868285841412014 2loss/SRqqt4/assignment_loss‚0.9985402896814048 /loss/FRt3/detection_loss‚1.7602444512440878 %loss/total_loss‚43.30767220957044 epoch‚8 /loss/FRt4/detection_loss‚2.3659164167112774 /loss/FBt3/detection_loss‚0.9524323916950919 _runtime‚ 7995.04077384 _step‚843 0loss/FRt2/assignment_loss‚2.7726414483420703 0loss/SRqqt2/detection_loss‚1.875516095960682 1loss/SRqqt4/detection_loss‚1.7879429353773595 trainer/global_step‚10249 /loss/FBt1/detection_loss‚2.1839555452267327 % _timestamp‚1.7884128920534081e+09 2loss/SRqqt1/assignment_loss‚1.7306206102882113 2loss/SRqqt3/assignment_loss‚0.7669822916346178 1loss/FBt2/assignment_loss‚0.36717251657197875 0loss/FBt3/assignment_loss‚0.3676172031295656 1loss/SRqqt1/detection_loss‚2.3460889645985197 /loss/FBt2/detection_loss‚3.7868451277414956 2loss/SRqqt2/assignment_loss‚1.3747328022833576 /loss/FRt1/detection_loss‚2.8847751716772714 0loss/SRqqt3/detection_loss‚0.869011581282724 0loss/FRt1/assignment_loss‚2.5725013290842376 0loss/FRt3/assignment_loss‚1.6589968473303551 1loss/FBt4/assignment_loss‚0.05063441935067203 .loss/FBt4/detection_loss‚4.277999838193257 1loss/FBt1/assignment_loss‚0.22376626854061676 .loss/FRt2/detection_loss‚2.145909079483649ËJå7ƒ&‚(:’ ë‡äÔ—¶„ gpu.0.gpu‚28.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.611' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.322200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DÖló~„&‚(:ó ë‡äÔ¯Çô  disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.drbd2080.out‚0# disk.nvme0c0n1.in‚ 1187.62109375" proc.memory.rssMB‚ 2810.9296875* disk./.usagePercent‚16.605336307062235" disk.dm-0.out‚53117.1376953125 network.sent‚ 2.503544e+07 network.recv‚ 2.949138e+06 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375 disk.dm-1.out‚ 367.68359375 disk.sda.in‚ 0.1953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0$ memory_percent‚82.44423866271973( proc.memory.availableMB‚ 2876.3359375* proc.memory.percent‚17.156553268432617% disk./.usageGB‚247.43151092529297% disk.nvme0n1.out‚53117.1376953125 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 367.68359375 disk.sda3.out‚ 367.68359375 disk.drbd2266.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚53117.1376953125 disk.rbd1.out‚ 34.896484375 cpu‚25.34741415013687 proc.cpu.threads‚17 disk.drbd2265.in‚0! disk.nvme0n1.in‚ 1187.62109375ô3`’…&‚(:‡ ú‡äÔ¹½ð gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.142' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pü±Ì}†&‚(:ò ú‡äÔ”žˆ  disk.dm-0.in‚ 1187.62109375 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0) proc.memory.availableMB‚ 2876.33984375 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 368.5703125 network.sent‚ 2.5133511e+07% disk.nvme0n1.out‚53180.3720703125 disk.drbd2265.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0$ memory_percent‚82.44421482086182 cpu‚25.346827844407677 disk.dm-1.out‚ 368.5703125 disk.drbd2266.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2810.9296875" disk.dm-0.out‚53180.3720703125 disk.sda.out‚ 368.5703125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.rbd0.in‚0 network.recv‚ 2.955647e+06* disk./.usagePercent‚16.606466580325908$ disk./.usageGB‚247.4483528137207 disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1187.62109375 disk.sda3.in‚ 0.1953125' disk.nvme0c0n1.out‚53180.3720703125 disk.rbd1.out‚ 34.896484375 disk.drbd2080.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.156553268432617ûˆ‰2‡&- _wandbruntime‚8031  _runtime‚8031‹ãZȈ& & lr-AdamW/pg1‚1.798126491823446e-05 & lr-AdamW/pg2‚1.798126491823446e-05 trainer/global_step‚10299 % _timestamp‚1.7884129282405777e+09 _runtime‚8031.228257015 _step‚844}z%½Í‰&Ç & lr-AdamW/pg1‚1.798126491823446e-05 & lr-AdamW/pg2‚1.798126491823446e-05 trainer/global_step‚10299 % _timestamp‚1.7884129282405777e+09 _runtime‚8031.228257015 _step‚844̘Ä÷2Š&- _wandbruntime‚8031  _runtime‚8031!Ÿd}è'‹&âO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149818620 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654112598996480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149818620 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚599015547135488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚599015547135488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _step‚845 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194476858 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚607643188337152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344295480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8031.229273202 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚583316282938368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599015529132544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149818622 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚10299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49243857 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87414116 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8627641201664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8595878264798 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607643188337152 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136658421 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654112622936576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607642513971712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚583316299977728 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884129282415037e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚607642513971712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9050099583488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8626984839168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚663162722520064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚591911506765410 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844?€nÔŒ&çO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 Bƒ3Xª%DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607642513971712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚583316299977728 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884129282415037e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚607642513971712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9050099583488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8626984839168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚663162722520064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚591911506765410 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149818620 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654112598996480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149818620 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚599015547135488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚599015547135488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _step‚845 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194476858 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚607643188337152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344295480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8031.229273202 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚583316282938368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599015529132544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149818622 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚10299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49243857 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87414116 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8627641201664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8595878264798 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607643188337152 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136658421 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654112622936576Ícήí2&- _wandbruntime‚8031  _runtime‚8031ÝÓ¹¥ò&Ž&ìM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8628460642816 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚599071900174336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8628460642816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚583371187068928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87422286 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194475274 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚599071882171392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87422284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49248333 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9050791234560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚607700360817152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654174249734144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149832636 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663225232713728 _runtime‚8031.917114417 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194495338 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344327976 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136671071 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344327976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8627804267008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49248785 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344307910 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591967898808258 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136670617 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607699686438400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194475274 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452VL a÷&&ñM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136670617 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607699686438400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194475274 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8628460642816 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚599071900174336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8628460642816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚583371187068928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87422286 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194475274 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚599071882171392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87422284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49248333 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9050791234560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚607700360817152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654174249734144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149832636 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663225232713728 _runtime‚8031.917114417 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194495338 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344327976 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136671071 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344327976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8627804267008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49248785 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344307910 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591967898808258 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610501151719936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8636059166450 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610501151719936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586061489466368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150519422 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚594697548632818 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8668627007488 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195402910 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195382872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87822518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657195135879680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922332 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586061506505728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9093019154944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116Ê4A)ì'œ&æO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137295786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚849 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137296211 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8667970644992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150519422 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚601833181074944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚601833181074944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87822516 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666288178974720 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195402910 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195382872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87822518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657195135879680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922332 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586061506505728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9093019154944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 trainer/global_step‚10349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195382872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚601833199077888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49473270 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788412965326467e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610501151719936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8636059166450 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610501151719936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586061489466368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150519422 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚594697548632818 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8668627007488 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8669446448640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚586116393596928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150533436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195401326 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150533436 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49477783 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666350696767488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601889534113792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150533438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195401326 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8636875601414 trainer/global_step‚10349 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137308467 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚610558324186624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954828 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657256786617344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657256810557440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87830684 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚594753940675666 _runtime‚ 8069.00631737 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8637530039378 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345934762 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚610558998565376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150533438 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137308903 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586116410636288 _step‚850 H:DeviceStatsMonitor.on_train_batch_ª¿‰¨end/active.all.allocated‚ 345954828 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚610558998565376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0·®3Tö&Ÿ&ðM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657256810557440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87830684 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚594753940675666 _runtime‚ 8069.00631737 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8637530039378 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345934762 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚610558998565376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150533438 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137308903 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586116410636288 _step‚850 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345954828 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚610558998565376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49478217 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8669446448640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚586116393596928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150533436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195401326 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150533436 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49477783 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666350696767488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601889534113792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150533438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195401326 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8636875601414 trainer/global_step‚10349 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137308467 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚610558324186624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954828 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657256786617344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430Òº©¸2 &- _wandbruntime‚8069  _runtime‚8069 ËdAa¡&Û 2loss/SRqqt2/assignment_loss‚1.3875542143014845 /loss/FBt4/detection_loss‚2.1955651342868805 0loss/FRt1/assignment_loss‚2.5015894532203675 3loss/FBt1/assignment_loss‚0.0014173254021443427 1loss/SRqqt2/detection_loss‚1.8451288025136348 $ _timestamp‚1.788412966020141e+09 1loss/SRqqt1/assignment_loss‚1.566601320528067 /loss/FRt4/detection_loss‚2.5809018740782865 trainer/global_step‚10349 /loss/FRt4/assignment_loss‚2.327478334710405 2loss/SRqqt3/assignment_loss‚0.9129525741358122 /loss/FRt3/detection_loss‚1.9126496916312887 1loss/SRqqt4/detection_loss‚1.7031944262981416 _step‚851 0loss/FRt3/assignment_loss‚1.6873068602188774 .loss/FBt1/detection_loss‚5.334178984165192 1loss/FBt3/assignment_loss‚0.11895587657600996 .loss/FRt2/detection_loss‚2.408640934808996 /loss/FBt2/detection_loss‚2.1928365495469837 /loss/FBt3/detection_loss‚0.6287351994996708 epoch‚8 0loss/FRt2/assignment_loss‚2.6336013175345756 2loss/SRqqt4/assignment_loss‚1.4310115570388735 1loss/FBt4/assignment_loss‚0.17324029160226928 %loss/total_loss‚41.32765960010791 1loss/FBt2/assignment_loss‚0.07814584417630815 /loss/FRt1/detection_loss‚2.6561879595120748 1loss/SRqqt1/detection_loss‚2.3270844725462108 1loss/SRqqt3/detection_loss‚0.7227006017753522 _runtime‚8069.007514339(j Wf¢&à 0loss/FRt3/assignment_loss‚1.6873068602188774 .loss/FBt1/detection_loss‚5.334178984165192 1loss/FBt3/assignment_loss‚0.11895587657600996 .loss/FRt2/detection_loss‚2.408640934808996 /loss/FBt2/detection_loss‚2.1928365495469837 /loss/FBt3/detection_loss‚0.6287351994996708 epoch‚8 0loss/FRt2/assignment_loss‚2.6336013175345756 2loss/SRqqt4/assignment_loss‚1.4310115570388735 1loss/FBt4/assignment_loss‚0.17324029160226928 %loss/total_loss‚41.32765960010791 1loss/FBt2/assignment_loss‚0.07814584417630815 /loss/FRt1/detection_loss‚2.6561879595120748 1loss/SRqqt1/detection_loss‚2.3270844725462108 1loss/SRqqt3/detection_loss‚0.7227006017753522 _runtime‚8069.007514339 2loss/SRqqt2/assignment_loss‚1.3875542143014845 /loss/FBt4/detection_loss‚2.1955651342868805 0loss/FRt1/assignment_loss‚2.5015894532203675 3loss/FBt1/assignment_loss‚0.0014173254021443427 1loss/SRqqt2/detection_loss‚1.8451288025136348 $ _timestamp‚1.788412966020141e+09 1loss/SRqqt1/assignment_loss‚1.566601320528067 /loss/FRt4/detection_loss‚2.5809018740782865 trainer/global_step‚10349 /loss/FRt4/assignment_loss‚2.327478334710405 2loss/SRqqt3/assignment_loss‚0.9129525741358122 /loss/FRt3/detection_loss‚1.9126496916312887 1loss/SRqqt4/detection_loss‚1.7031944262981416 _step‚851Ó4”^d’£&‚(:‡ §ˆäÔÄ„Ø  gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.621' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s{Éë¤&‚(:ô §ˆäÔúª×  disk.dm-1.in‚ 0.1953125 disk.sda2.in‚0$ memory_percent‚82.44717121124268" proc.memory.rssMB‚ 2810.9296875 disk.drbd2265.out‚0 disk.sda.out‚ 370.09765625! disk.nvme0n1.in‚ 1187.62109375* proc.memory.percent‚17.156553268432617 disk.rbd1.out‚ 34.896484375 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 370.09765625 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚53377.9033203125 network.recv‚ 2.970444e+06$ disk./.usageGB‚247.4958839416504 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚53377.9033203125 cpu‚25.345173617370122) disk./.usagePercent‚16.60965643420254 disk.sda3.out‚ 370.09765625 disk.drbd2266.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 network.sent‚ 2.5253694e+07 disk.drbd2080.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚53377.9033203125 disk.dm-0.in‚ 1187.62109375) proc.memory.availableMB‚ 2875.85546875a-´¦¥&‚(:’ ¶ˆäÔ’¦ü gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.696' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.139199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±rÍz¦&‚(:ï ¶ˆäÔׂÒ' disk.nvme0c0n1.out‚53440.0556640625! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.sda3.out‚ 370.828125 disk.drbd2080.in‚0$ memory_percent‚82.44726657867432 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 34.896484375" disk.dm-0.out‚53440.0556640625 disk.rbd0.in‚0% disk.nvme0n1.out‚53440.0556640625 disk.dm-1.out‚ 370.828125 disk.drbd2265.out‚0 cpu‚25.344641066362183 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 disk.rbd0.out‚0$ disk./.usageGB‚247.5156707763672 disk.drbd2265.in‚0 disk.sda.out‚ 370.828125 disk.drbd2080.out‚0 network.sent‚ 2.5361796e+07 network.recv‚ 2.977628e+06# disk.nvme0c0n1.in‚ 1187.62109375* disk./.usagePercent‚16.610984345282645 disk.rbd1.in‚0 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125) proc.memory.availableMB‚ 2875.83984375" proc.memory.rssMB‚ 2810.9296875* proc.memory.percent‚17.156553268432617‰$pÔœ§&‚(:‘ ňäÔ¬»ý gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.414' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.68279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–QRTy¨&‚(:î ňäÔ‘Ð  disk.drbd1716.out‚0 disk.rbd1.out‚ 34.896484375! disk.nvme0n1.in‚ 1187.62109375 network.recv‚ 2.98213e+06 disk.drbd2266.in‚0 disk.drbd1716.in‚0) proc.memory.availableMB‚ 2875.83984375* proc.memory.percent‚17.156553268432617 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 cpu‚25.344087640954275 network.sent‚ 2.5368306e+07 disk.drbd2080.in‚0 disk.sda.out‚ 371.4765625 proc.cpu.threads‚17% disk./.usageGB‚247.52689361572266 disk.drbd2266.out‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚53493.068359375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0! disk.dm-0.out‚53493.068359375 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 371.4765625" proc.memory.rssMB‚ 2810.9296875$ disk.nvme0n1.out‚53493.068359375 disk.dm-0.in‚ 1187.62109375 disk.dm-1.out‚ 371.4765625 disk.drbd2265.out‚0$ memory_percent‚82.44726657867432) disk./.usagePercent‚16.61173751944838 disk.drbd2080.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2265.in‚0À6 X2©&- _wandbruntime‚8105  _runtime‚8105Þmɗʪ&Ä _runtime‚8105.496292346 _step‚852 ' lr-AdamW/pg1‚1.6068348127858788e-05 ' lr-AdamW/pg2‚1.6068348127858788e-05 trainer/global_step‚10399 % _timestamp‚1.7884130025088763e+09oP¤Ï«&É _step‚852 ' lr-AdamW/pg1‚1.6068348127858788e-05 ' lr-AdamW/pg2‚1.6068348127858788e-05 trainer/global_step‚10399 % _timestamp‚1.7884130025088763e+09 _runtime‚8105.496292346Ô\¯¡‡2¬&- _wandbruntime‚8105  _runtime‚8105áÎ^ºæ'­&àO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597484261977342 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604650851020288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9135876586496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604650833017344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588806713033728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604650851020288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49703634 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151220220 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137934552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597483590500226 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151220222 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588806695994368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49703187 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604650833017344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151220222 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8677548943614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88230918 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚669413549349376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚853 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347529144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚669413747399168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚ 8105.4973177 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196308924 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8708956450816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347549184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613360463833600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613360463833600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613359789468160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8709612813312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88230916 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚660277696702976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613359789468160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8708956450816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8709612813312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137934103 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 trainer/global_step‚10399 % _timestamp‚1.7884130025096197e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196328962 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196308924 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9136050696192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8676894505858˜ ÛÒ ®&åO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196328962 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196308924 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9136050696192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8676894505858 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597484261977342 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604650851020288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9135876586496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604650833017344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588806713033728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604650851020288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49703634 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151220220 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137934552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597483590500226 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151220222 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_splitº²F¦.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588806695994368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49703187 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604650833017344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151220222 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8677548943614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88230918 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚669413549349376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚853 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347529144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚669413747399168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚ 8105.4973177 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196308924 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8708956450816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347549184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613360463833600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613360463833600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613359789468160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8709612813312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88230916 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚660277696702976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613359789468160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8708956450816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8709612813312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137934103 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 trainer/global_step‚10399 % _timestamp‚1.7884130025096197e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952ÕXĘû2¯&- _wandbruntime‚8106  _runtime‚8106È©¨Wò&°&ìM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8106.196348218 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597539982543074 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151234238 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚669476052935168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚660339323500544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561614 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88239086 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151234236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8709775878656 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884130032081933e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137947303 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604707204059136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8677710940822 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚10399 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613417636313600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88239084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49708217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8709775878656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196327378 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137946856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669476246777344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196327378 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597539311065750 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _step‚854 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8710432254464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8678365378786 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151234236 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚613417636313600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347581680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚613416961934848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚660339347440640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588861600124928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588861617164288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49707772 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9136729434624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928,÷&±&ñM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8678365378786 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151234236 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚613417636313600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347581680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚613416961934848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚660339347440640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588861600124928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588861617164288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49707772 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9136729434624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8106.196348218 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597539982543074 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151234238 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚669476052935168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚660339323500544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561614 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88239086 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151234236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8709775878656 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884130032081933e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137947303 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604707204059136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8677710940822 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚10399 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613417636313600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88239084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49708217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8709775878656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196327378 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137946856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669476246777344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196327378 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597539311065750 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _step‚854 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8710432254464Ö§ìk2²&- _wandbruntime‚8106  _runtime‚8106µ¢ èZ³&Ô 0loss/FRt3/assignment_loss‚1.2683572642346645 0loss/FRt4/assignment_loss‚2.5140004416681685 1loss/FBt1/assignment_loss‚0.04276385900539026 epoch‚8 /loss/FRt1/assignment_loss‚3.330862087361953 /loss/FRt4/detection_loss‚2.1793720947121673 0loss/SRqqt1/detection_loss‚2.389549854923697 1loss/SRqqt4/detection_loss‚1.6886671537258586 /loss/FRt2/detection_loss‚2.1923037319808723 /loss/FBt3/detection_loss‚0.5214591712622411 trainer/global_step‚10399 /loss/FRt2/assignment_loss‚3.021388699114323 2loss/SRqqt2/assignment_loss‚1.4551054801283911 2loss/SRqqt4/assignment_loss‚1.1229492907000125 /loss/FBt2/detection_loss‚3.3021847280589016 .loss/FBt4/detection_loss‚2.651055008172989 _runtime‚8106.197882223 2loss/SRqqt1/assignment_loss‚1.3558710403740406 1loss/FBt4/assignment_loss‚0.08134552298967417 -loss/FRt3/detection_loss‚1.55096973526351 _step‚855 2loss/SRqqt3/assignment_loss‚0.9662278911483655 1loss/FBt3/assignment_loss‚0.09951258567230137 1loss/SRqqt3/detection_loss‚0.8561059248685423 $loss/total_loss‚40.2002737607901 1loss/FBt2/assignment_loss‚0.09392281934906971 /loss/FRt1/detection_loss‚2.8235894187408337 1loss/SRqqt2/detection_loss‚1.7507743286815556 /loss/FBt1/detection_loss‚2.9419356286525726 " _timestamp‚1.7884130032105e+09Iur_´&Ù /loss/FRt2/assignment_loss‚3.021388699114323 2loss/SRqqt2/assignment_loss‚1.4551054801283911 2loss/SRqqt4/assignment_loss‚1.1229492907000125 /loss/FBt2/detection_loss‚3.3021847280589016 .loss/FBt4/detection_loss‚2.651055008172989 _runtime‚8106.197882223 2loss/SRqqt1/assignment_loss‚1.3558710403740406 1loss/FBt4/assignment_loss‚0.08134552298967417 -loss/FRt3/detection_loss‚1.55096973526351 _step‚855 2loss/SRqqt3/assignment_loss‚0.9662278911483655 1loss/FBt3/assignment_loss‚0.09951258567230137 1loss/SRqqt3/detection_loss‚0.8561059248685423 $loss/total_loss‚40.2002737607901 1loss/FBt2/assignment_loss‚0.09392281934906971 /loss/FRt1/detection_loss‚2.8235894187408337 1loss/SRqqt2/detection_loss‚1.7507743286815556 /loss/FBt1/detection_loss‚2.9419356286525726 " _timestamp‚1.7884130032105e+09 0loss/FRt3/assignment_loss‚1.2683572642346645 0loss/FRt4/assignment_loss‚2.5140004416681685 1loss/FBt1/assignment_loss‚0.04276385900539026 epoch‚8 /loss/FRt1/assignment_loss‚3.330862087361953 /loss/FRt4/detection_loss‚2.1793720947121673 0loss/SRqqt1/detection_loss‚2.389549854923697 1loss/SRqqt4/detection_loss‚1.6886671537258586 /loss/FRt2/detection_loss‚2.1923037319808723 /loss/FBt3/detection_loss‚0.5214591712622411 trainer/global_step‚10399×õ6k’µ&‚(:‡ ÔˆäÔ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0뙶&‚(:ö ÔˆäÔ¿¥  disk.drbd2265.in‚0 cpu‚25.343520211096497) disk./.usagePercent‚16.61248557346343* proc.memory.percent‚17.156553268432617 network.sent‚ 2.5373644e+07 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2810.9296875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚53545.99169921875 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 372.05859375 disk.drbd2080.in‚0 disk.drbd1716.out‚0# disk.dm-0.out‚53545.99169921875$ memory_percent‚82.44748115539551 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0( disk.nvme0c0n1.out‚53545.99169921875! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.dm-1.out‚ 372.05859375 disk.drbd2265.out‚0( proc.memory.availableMB‚ 2875.8046875 network.recv‚ 2.985716e+06 disk.sda.out‚ 372.05859375 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚247.5380401611328 disk.sda1.out‚0 disk.rbd1.out‚ 34.982421875ÞÍÇ’·&‚(:‡ ãˆäÔ晃 gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ñêׄ¸&‚(:ù ãˆäÔƘˆ  disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd1716.in‚0) proc.memory.availableMB‚ 2875.79296875 disk.rbd1.in‚0 disk.dm-1.out‚ 372.45703125$ memory_percent‚82.44755268096924* disk./.usagePercent‚16.613411552714776 disk.drbd2265.out‚0 disk.sda.out‚ 372.45703125 disk.drbd2080.out‚0* proc.memory.percent‚17.156553268432617 disk.sda3.in‚ 0.1953125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1187.62109375! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2266.out‚0 network.sent‚ 2.5482385e+07% disk./.usageGB‚247.55183792114258 disk.rbd0.out‚0" proc.memory.rssMB‚ 2810.9296875 cpu‚25.342975389439953 disk.rbd1.out‚ 35.044921875# disk.dm-0.out‚53605.48583984375 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 network.recv‚ 2.992491e+06 disk.dm-0.in‚ 1187.62109375 disk.sda3.out‚ 372.45703125 disk.sda1.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚53605.48583984375 proc.cpu.threads‚17& disk.nvme0n1.out‚53605.48583984375 disk.drbd2265.in‚0B ÿ2¹&- _wandbruntime‚8141  _runtime‚8141ÙÏô+º&Ä _step‚856 ' lr-AdamW/pg1‚1.5^ÜÃ÷Ÿ149759869723452e-05 ' lr-AdamW/pg2‚1.5149759869723452e-05 trainer/global_step‚10449 % _timestamp‚1.7884130389649043e+09 _runtime‚8141.952351478-ôŸÏ»&É ' lr-AdamW/pg2‚1.5149759869723452e-05 trainer/global_step‚10449 % _timestamp‚1.7884130389649043e+09 _runtime‚8141.952351478 _step‚856 ' lr-AdamW/pg1‚1.5149759869723452e-05ØÞ 2¼&- _wandbruntime‚8141  _runtime‚81415Bè'½&âO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8750598619136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚616219101581824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9179021147136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138572577 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚672539077655040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚607468502962688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚8141.953330015 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138572132 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88639318 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151921022 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349155996 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚607468484959744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8750598619136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚607468502962688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8718384283022 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197234976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88639316 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197234976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591551902522368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8749942256640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚607468484959744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672539254733312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591551919561728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616218427216384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600269632367634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚616219101581824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884130389656515e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151921020 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49933259 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663360209646080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚616218427216384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349176036 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151921020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49932816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚10449|\ Ôí'¾&çO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197234976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591551902522368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8749942256640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚607468484959744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672539254733312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591551919561728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616218427216384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600269632367634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚616219101581824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884130389656515e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151921020 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49933259 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663360209646080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚616218427216384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349176036 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151921020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49932816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚10449 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8750598619136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚616219101581824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9179021147136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138572577 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚672539077655040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚607468502962688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚8141.953330015 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138572132 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88639318 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151921022 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349155996 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚607468484959744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8750598619136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚607468502962688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8718384283022 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197234976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88639316 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430ÙÚö2\2¿&- _wandbruntime‚8142  _runtime‚8142cÖÚoð&À&êM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751418060288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9179888561664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 # _timestamp‚1.78841303964697e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138584880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49937834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151935038 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚858 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚607524837998592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚607524856001536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8750761684480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672601772885504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8719200718194 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88647484 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349188466 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197273494 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88647486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600326024410482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607524856001536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49937396 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197253430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349208532 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚591606806652928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚8142.634948392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138585320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚10449 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚607524837998592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349188466 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚600325352933158 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751418060288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151935038 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197273494 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚616275599683072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚672601585334784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151935036 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚616276274061824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚591606823692288êD¤†1Á&ïM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151935038 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚858 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚607524837998592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚607524856001536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8750761684480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672601772885504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested4¢öºÄ_bytes.small_pool.current‚ 654437964 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8719200718194 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88647484 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349188466 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197273494 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88647486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600326024410482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607524856001536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49937396 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197253430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349208532 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚591606806652928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚8142.634948392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138585320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚10449 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚607524837998592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349188466 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚600325352933158 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751418060288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151935038 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197273494 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚616275599683072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚672601585334784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151935036 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚616276274061824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚591606823692288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751418060288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9179888561664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 # _timestamp‚1.78841303964697e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138584880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49937834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ú©£_2Â&- _wandbruntime‚8142  _runtime‚8142ª|d°`Ã&Ú 0loss/FRt1/assignment_loss‚3.6291241162532084 0loss/FRt4/assignment_loss‚1.9500789940357208 2loss/SRqqt2/assignment_loss‚1.3278237826683943 1loss/SRqqt4/assignment_loss‚1.241957575824255 /loss/FRt3/detection_loss‚1.5883478909941777 /loss/FRt4/detection_loss‚2.3812499387483848 % _timestamp‚1.7884130396485558e+09 1loss/FBt3/assignment_loss‚0.13583505703684154 1loss/SRqqt3/detection_loss‚0.8727549554314464 1loss/SRqqt4/detection_loss‚1.5041704773902893 1loss/SRqqt2/detection_loss‚1.8978940336143268 _step‚859 /loss/FRt3/assignment_loss‚1.305099348976927 2loss/SRqqt3/assignment_loss‚0.6520529817789793 2loss/FBt2/assignment_loss‚0.011549998167902231 .loss/FRt2/detection_loss‚2.400784715643883 1loss/SRqqt1/detection_loss‚2.6595810008049012 .loss/FBt2/detection_loss‚3.035172939300537 /loss/FBt3/detection_loss‚0.4636843062998891 /loss/FBt4/detection_loss‚2.3509603568485806 .loss/FRt1/detection_loss‚2.810477491971609 2loss/SRqqt1/assignment_loss‚1.9091134068369866 %loss/total_loss‚39.58087080206164 0loss/FRt2/assignment_loss‚2.3692305714656143 1loss/FBt1/assignment_loss‚0.21033751763861802 /loss/FBt1/detection_loss‚2.8348230004310606 epoch‚8 trainer/global_step‚10449 _runtime‚8142.635947327 1loss/FBt4/assignment_loss‚0.03876634389910448þÅëeÄ&ß 2loss/SRqqt3/assignment_loss‚0.6520529817789793 2loss/FBt2/assignment_loss‚0.011549998167902231 .loss/FRt2/detection_loss‚2.400784715643883 1loss/SRqqt1/detection_loss‚2.6595810008049012 .loss/FBt2/detection_loss‚3.035172939300537 /loss/FBt3/detection_loss‚0.4636843062998891 /loss/FBt4/detection_loss‚2.3509603568485806 .loss/FRt1/detection_loss‚2.810477491971609 2loss/SRqqt1/assignment_loss‚1.9091134068369866 %loss/total_loss‚39.58087080206164 0loss/FRt2/assignment_loss‚2.3692305714656143 1loss/FBt1/assignment_loss‚0.21033751763861802 /loss/FBt1/detection_loss‚2.8348230004310606 epoch‚8 trainer/global_step‚10449 _runtime‚8142.635947327 1loss/FBt4/assignment_loss‚0.03876634389910448 0loss/FRt1/assignment_loss‚3.6291241162532084 0loss/FRt4/assignment_loss‚1.9500789940357208 2loss/SRqqt2/assignment_loss‚1.3278237826683943 1loss/SRqqt4/assignment_loss‚1.241957575824255 /loss/FRt3/detection_loss‚1.5883478909941777 /loss/FRt4/detection_loss‚2.3812499387483848 % _timestamp‚1.7884130396485558e+09 1loss/FBt3/assignment_loss‚0.13583505703684154 1loss/SRqqt3/detection_loss‚0.8727549554314464 1loss/SRqqt4/detection_loss‚1.5041704773902893 1loss/SRqqt2/detection_loss‚1.8978940336143268 _step‚859 /loss/FRt3/assignment_loss‚1.305099348976927Û_þ’Å&‚(:‡ òˆäÔž« gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÂ¥ƒÆ&‚(:ø òˆäÔï² % disk./.usageGB‚247.52946853637695 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375# disk.dm-0.out‚53663.26318359375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2080.out‚0 cpu‚25.34238054911353 disk.rbd1.in‚0 disk.rbd1.out‚ 35.044921875" proc.memory.rssMB‚ 2810.9296875 network.sent‚ 2.5488267e+07 network.recv‚ 2.996814e+06 disk.sda1.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚53663.26318359375 disk.sda.out‚ 372.93359375 disk.sda3.out‚ 372.93359375 disk.drbd2080.in‚0* disk./.usagePercent‚16.611910324533994 disk.dm-0.in‚ 1187.62109375 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 372.93359375 disk.sda3.in‚ 0.1953125) proc.memory.availableMB‚ 2875.91015625 disk.drbd1716.in‚0& disk.nvme0n1.out‚53663.26318359375 disk.dm-1.in‚ 0.1953125$ memory_percent‚82.44683742523193* proc.memory.percent‚17.156553268432617 disk.rbd0.out‚0ÙüRÇ&‚(:… ‰äÔ­žˆ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.38' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ[EçÈ&‚(:ô ‰äÔˆÎð disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2810.9296875 network.sent‚ 2.5597339e+07 disk.dm-0.in‚ 1187.62109375 disk.rbd0.out‚0 disk.dm-1.out‚ 373.80859375 disk.sda.out‚ 373.80859375) proc.memory.availableMB‚ 2895.00390625 network.recv‚ 3.003972e+06( disk./.usagePercent‚16.6133997763682 disk.sda1.in‚0 disk.rbd1.out‚ 35.0771484375% disk.nvme0n1.out‚53733.1083984375 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375' disk.nvme0c0n1.out‚53733.1083984375$ memory_percent‚82.33029842376709% disk./.usageGB‚247.55166244506836" disk.dm-0.out‚53733.1083984375 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0* proc.memory.percent‚17.156553268432617 disk.rbd0.in‚0 disk.sda3.in‚ 0.1953125 cpu‚25.34184808240425 disk.sda1.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.sda3.out‚ 373.80859375 proc.cpu.threads‚17 disk.drbd2080.in‚0x*vxÉ&‚(:’ ‰äÔºå± gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.737' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.547399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NM}Ê&‚(:ö ‰äÔ¬“¡  disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.in‚0" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd1.out‚ 35.0771484375$ memory_percent‚82.33036994934082 network.sent‚ 2.5602634e+07% disk./.usageGB‚247.56695938110352 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 374.3984375( proc.memory.availableMB‚ 2894.9921875! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚53786.66064453125 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 network.recv‚ 3.007539e+06 disk.sda1.in‚0( disk.nvme0c0n1.out‚53786.66064453125 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 374.3984375 disk.drbd2265.in‚0 disk.sda.out‚ 374.3984375 disk.sda2.out‚0# disk.dm-0.out‚53786.66064453125 disk.dm-0.in‚ 1187.62109375* disk./.usagePercent‚16.614426366580506 disk.drbd2080.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚17.156553268432617 cpu‚25.341311749608472¤Áï–2Ë&- _wandbruntime‚8178  _runtime‚8178FÂYÇÌ&Á _step‚860 & lr-AdamW/pg1‚1.425667537119506e-05 & lr-AdamW/pg2‚1.425667537119506e-05 trainer/global_step‚10499 $ _timestamp‚1.788413075806889e+09 _runtime‚8178.794306505ë>[;ÌÍ&Æ & lr-AdamW/pg1‚1.425667537119506e-05 & lr-AdamW/pg2‚1.425667537119506e-05 trainer/global_step‚10499 $ _timestamp‚1.788413075806889e+09 _runtime‚8178.794306505 _step‚860Ü8XDI2Î&- _wandbruntime‚8178  _runtime‚8178F‹ƒÔç'Ï&áO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675664606669312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350802888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9221860140032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152621820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139207618 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89047718 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198161028 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619077064964608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚603056345712158 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚594297126089728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚10499 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675664792136192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619077739330048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚610286154905088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50159902 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8791584424960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50160344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152621822 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚610286136902144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8759219622430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8790928062464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89047716 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152621820 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675664606669312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350802888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9221860140032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152621820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139207618 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89047718 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198161028 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619077064964608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚603056345712158 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚594297126089728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚10499 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675664792136192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619077739330048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚610286154905088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50159902 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8791584424960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50160344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152621822 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚610286136902144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8759219622430Ýp#x2Ñ&- _wandbruntime‚8179  _runtime‚8179GLîXò&Ò&ìM BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139220317 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350815318 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚862 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚619134237431296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8792403866112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8759381619638 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198179482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139220754 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50164868 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚603111394800566 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89055886 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚594352030220288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚610342507943936 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚8179.488127894 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8792403866112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚619134911810048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152635838 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675727303611904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9222716697088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152635836 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198179482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152635836 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8760036057602 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8791747490304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675727113964032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚610342507943936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89055884 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430,w?÷&Ó&ñM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198179482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139220754 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50164868 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚603111394800566 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89055886 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚594352030220288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚610342507943936 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚8179.488127894 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8792403866112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚619134911810048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152635838 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675727303611904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9222716697088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152635836 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198179482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152635836 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8760036057602 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8791747490304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675727113964032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚610342507943936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89055884 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139220317 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350815318 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚862 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚619134237431296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8792403866112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8759381619638Þ캽§2Ô&- _wandbruntime‚8179  _runtime‚8179Ý3ÆbÕ&Ü 1loss/FBt2/assignment_loss‚0.04705264749160657 2loss/FBt4/assignment_loss‚0.007427979438944021 epoch‚8 trainer/global_step‚10499 /loss/FRt1/detection_loss‚3.1101502726475396 /loss/FRt3/detection_loss‚1.4352600142726155 1loss/SRqqt1/detection_loss‚1.9467687994241714 /loss/FBt2/detection_loss‚3.2817425529162088 _runtime‚8179.489262595 2loss/SRqqt4/assignment_loss‚1.2169232792644338 /loss/FRt2/detection_loss‚2.3911764843519343 /loss/FBt3/detection_loss‚0.5994125322085693 /loss/FBt4/detection_loss‚2.5529206469655037 $ _timestamp‚1.788413076501899e+09 1loss/SRqqt1/assignment_loss‚1.596088855760172 1loss/SRqqt3/assignment_loss‚0.777312211360642 1loss/SRqqt2/detection_loss‚1.8337173950164876 1loss/SRqqt3/detection_loss‚0.8021368017449738 /loss/FBt1/detection_loss‚2.7782749772071837 0loss/FRt1/assignment_loss‚2.6455677102009454 0loss/FRt3/assignment_loss‚1.5868229299295145 /loss/FRt4/detection_loss‚2.0804794342542183 0loss/SRqqt4/detection_loss‚1.819325448606502 2loss/FBt1/assignment_loss‚0.012868253607302904 1loss/FBt3/assignment_loss‚0.05886928481897989 _step‚863 0loss/FRt2/assignment_loss‚2.3739097589670224 2loss/SRqqt2/assignment_loss‚1.3117564770175105 &loss/total_loss‚38.850677475418664 /loss/FRt4/assignment_loss‚2.584712727945678)ÖŠÐgÖ&á /loss/FRt1/detection_loss‚3.1101502726475396 /loss/FRt3/detection_loss‚1.4352600142726155 1loss/SRqqt1/detection_loss‚1.9467687994241714 /loss/FBt2/detection_loss‚3.2817425529162088 _runtime‚8179.489262595 2loss/SRqqt4/assignment_loss‚1.2169232792644338 /loss/FRt2/detection_loss‚2.3911764843519343 /loss/FBt3/detection_loss‚0.5994125322085693 /loss/FBt4/detection_loss‚2.5529206469655037 $ _timestamp‚1.788413076501899e+09 1loss/SRqqt1/assignment_loss‚1.596088855760172 1loss/SRqqt3/assignment_loss‚0.777312211360642 1loss/SRqqt2/detection_loss‚1.8337173950164876 1loss/SRqqt3/detection_loss‚0.8021368017449738 /loss/FBt1/detection_loss‚2.7782749772071837 0loss/FRt1/assignment_loss‚2.6455677102009454 0loss/FRt3/assignment_loss‚1.5868229299295145 /loss/FRt4/detection_loss‚2.0804794342542183 0loss/SRqqt4/detection_loss‚1.819325448606502 2loss/FBt1/assignment_loss‚0.012868253607302904 1loss/FBt3/assignment_loss‚0.05886928481897989 _step‚863 0loss/FRt2/assignment_loss‚2.3739097589670224 2loss/SRqqt2/assignment_loss‚1.3117564770175105 &loss/total_loss‚38.850677475418664 /loss/FRt4/assignment_loss‚2.584712727945678 1loss/FBt2/assignment_loss‚0.04705264749160657 2loss/FBt4/assignment_loss‚0.007427979438944021 epoch‚8 trainer/global_step‚10499ßóË­“×&‚(:’ Ÿ‰äÔʸ› gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.296' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.459199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hoâ„Ø&‚(:ù Ÿ‰äÔéÈ   disk.sda2.out‚0 disk.dm-1.out‚ 374.79296875 disk.sda.in‚ 0.1953125 network.sent‚ 2.5608807e+07# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 disk.rbd1.out‚35.14013671875 disk.sda3.in‚ 0.1953125* proc.memory.percent‚17.156553268432617 network.recv‚ 3.011879e+06 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2810.9296875* disk./.usagePercent‚16.615777318338694 disk.sda1.in‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375# disk.dm-0.out‚53859.35791015625% disk./.usageGB‚247.58708953857422& disk.nvme0n1.out‚53859.35791015625 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.drbd2080.out‚0 disk.rbd0.out‚0# memory_percent‚82.3305606842041( disk.nvme0c0n1.out‚53859.35791015625 disk.sda.out‚ 374.79296875 disk.drbd2266.out‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 2894.9609375 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda3.out‚ 374.79296875 disk.rbd0.in‚0 cpu‚25.34077985695782398gÙ&‚(:’ ®‰äÔ½€ù gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.938' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.187599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ØG„Ú&‚(:ù ®‰äÔ˜ÁÓ network.sent‚ 2.5717574e+07 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚35.22607421875$ disk./.usageGB‚247.5938835144043 disk.sda.out‚ 375.38671875 disk.rbd1.in‚0$ memory_percent‚82.32903480529785* disk./.usagePercent‚16.616233267757174 disk.drbd2266.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚53913.20556640625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 375.38671875 disk.sda1.in‚0 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚53913.20556640625" proc.memory.rssMB‚ 2810.9296875 cpu‚25.340223394832933 proc.cpu.threads‚17* proc.memory.percent‚17.156553268432617 network.recv‚ 3.018977e+06 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375( proc.memory.availableMB‚ 2895.2109375 disk.dm-1.out‚ 375.38671875 disk.drbd2080.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚53913.20556640625Öé]2Û&- _wandbruntime‚8215  _runtime‚8215Øö‘ÉÜ&à ' lr-AdamW/pg1‚1.3389280749744538e-05 ' lr-AdamW/pg2‚1.3389280749744538e-05 trainer/global_step‚10549 % _timestamp‚1.7884131125525367e+09 _runtime‚ 8215.53997352 _step‚8649Ëâ[ÎÝ&È _runtime‚ 8215.53997352 _step‚864 ' lr-AdamW/pg1‚1.3389280749744538e-05 ' lr-AdamW/pg2‚1.3389280749744538e-05 trainer/global_step‚10549 % _timestamp‚1.7884131125525367e+09à OB‚2Þ&- _wandbruntime‚8215  _runtime‚82155UÑ ß&âO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597042332617728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚613103806847488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9264817957888 _runtime‚8215.540968818 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669525307352576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8831913868288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚865 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚613103788844544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_trai6ìUïân_batch_start/inactive_split.all.freed‚ 139845027 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8831913868288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153322620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50388911 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199107118 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199087080 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153322622 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9264989970432 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621935702712832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613103788844544 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8832570230784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678790297323008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚605841716102450 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚621936377078272 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139845484 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚621936377078272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669525283412480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚10549 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153322622 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚613103806847488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352429740 % _timestamp‚1.7884131125532653e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89456116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚605842387579566 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199107118 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50389366 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678790101370368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352409700 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89456118 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199087080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621935702712832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669525307352576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8831913868288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚865 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚613103788844544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139845027 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8831913868288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153322620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50388911 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199107118 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199087080 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153322622 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9264989970432 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621935702712832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613103788844544 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8832570230784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678790297323008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚605841716102450 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚621936377078272 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139845484 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚621936377078272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669525283412480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚10549 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153322622 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚613103806847488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352429740 % _timestamp‚1.7884131125532653e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89456116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚605842387579566 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199107118 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50389366 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678790101370368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352409700 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89456118 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199087080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621935702712832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89464286 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199125598 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50393918 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚613160159886336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚613160141883392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8833389671936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8832733296128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605897436667974 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚621993549558272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621993549558272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8800216959046 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199105534 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352442170 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 352442170 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199125598 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139857738 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9265835004416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9265660908032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597097219708928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚10549 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462236 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153336638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚669586958090240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚613160159886336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199105534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50393454 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139858204 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678852595058176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8833389671936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89464284 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669586934150144|Šmjêã&ñM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462236 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153336638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚669586958090240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚613160159886336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199105534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50393454 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139858204 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678852595058176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8833389671936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89464284 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669586934150144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚597097236748288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89464286 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocat´Ã…, ion.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199125598 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50393918 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚613160159886336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚613160141883392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8833389671936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8832733296128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605897436667974 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚621993549558272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621993549558272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8800216959046 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199105534 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352442170 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 352442170 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199125598 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139857738 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9265835004416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9265660908032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597097219708928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚10549âÃu¢2ä&- _wandbruntime‚8216  _runtime‚82166ï &[å&Õ 1loss/FBt3/assignment_loss‚0.14705233320197258 .loss/FRt2/detection_loss‚2.576956975460196 /loss/FRt4/detection_loss‚2.2606415858975164 %loss/total_loss‚41.55170691522892 /loss/FRt1/assignment_loss‚3.422511551691138 /loss/FRt2/assignment_loss‚2.599038542807102 0loss/FRt3/assignment_loss‚1.0807477998733521 2loss/SRqqt3/assignment_loss‚0.9103909686713576 1loss/SRqqt4/assignment_loss‚1.182998206457813 1loss/SRqqt4/detection_loss‚1.6563657802288179 .loss/FBt1/detection_loss‚2.783747739262051 /loss/FBt2/detection_loss‚3.0886884570121764 epoch‚8 .loss/FRt1/detection_loss‚3.233768051085265 1loss/SRqqt3/detection_loss‚1.1795564495862043 /loss/FBt4/detection_loss‚2.3620379037327237 2loss/SRqqt1/assignment_loss‚1.8070763916605048 /loss/FBt3/detection_loss‚0.6390634477138519 trainer/global_step‚10549 % _timestamp‚1.7884131132437127e+09 2loss/SRqqt2/assignment_loss‚1.6072956475394744 1loss/FBt1/assignment_loss‚0.14873511615830085 0loss/FBt2/assignment_loss‚0.5932212518506219 _runtime‚8216.231102791 0loss/FRt4/assignment_loss‚2.5555264475169004 0loss/FBt4/assignment_loss‚0.0422074926933015 /loss/FRt3/detection_loss‚1.7006358426064252 1loss/SRqqt2/detection_loss‚1.6114770427898124 0loss/SRqqt1/detection_loss‚2.361965889732043 _step‚867ðBg`æ&Ú /loss/FRt1/assignment_loss‚3.422511551691138 /loss/FRt2/assignment_loss‚2.599038542807102 0loss/FRt3/assignment_loss‚1.0807477998733521 2loss/SRqqt3/assignment_loss‚0.9103909686713576 1loss/SRqqt4/assignment_loss‚1.182998206457813 1loss/SRqqt4/detection_loss‚1.6563657802288179 .loss/FBt1/detection_loss‚2.783747739262051 /loss/FBt2/detection_loss‚3.0886884570121764 epoch‚8 .loss/FRt1/detection_loss‚3.233768051085265 1loss/SRqqt3/detection_loss‚1.1795564495862043 /loss/FBt4/detection_loss‚2.3620379037327237 2loss/SRqqt1/assignment_loss‚1.8070763916605048 /loss/FBt3/detection_loss‚0.6390634477138519 trainer/global_step‚10549 % _timestamp‚1.7884131132437127e+09 2loss/SRqqt2/assignment_loss‚1.6072956475394744 1loss/FBt1/assignment_loss‚0.14873511615830085 0loss/FBt2/assignment_loss‚0.5932212518506219 _runtime‚8216.231102791 0loss/FRt4/assignment_loss‚2.5555264475169004 0loss/FBt4/assignment_loss‚0.0422074926933015 /loss/FRt3/detection_loss‚1.7006358426064252 1loss/SRqqt2/detection_loss‚1.6114770427898124 0loss/SRqqt1/detection_loss‚2.361965889732043 _step‚867 1loss/FBt3/assignment_loss‚0.14705233320197258 .loss/FRt2/detection_loss‚2.576956975460196 /loss/FRt4/detection_loss‚2.2606415858975164 %loss/total_loss‚41.55170691522892ãgç²ö’ç&‚(:‡ ½‰äÔÆð— gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h/œ±„è&‚(:ù ½‰äÔ‹ôš # disk.nvme0c0n1.in‚ 1187.62109375 disk.dm-1.out‚ 376.02734375 proc.cpu.threads‚17& disk.nvme0n1.out‚53954.49658203125 network.sent‚ 2.5723793e+07 disk.sda.out‚ 376.02734375 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0$ disk./.usageGB‚247.6133918762207 disk.rbd0.out‚0( disk.nvme0c0n1.out‚53954.49658203125 disk.drbd1716.in‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 2894.8984375 cpu‚25.339722122377715* disk./.usagePercent‚16.617542490287278 disk.sda.in‚ 0.1953125 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125 disk.rbd0.in‚0 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚53954.49658203125$ memory_percent‚82.33094215393066" proc.memory.rssMB‚ 2810.9296875! disk.nvme0n1.in‚ 1187.62109375 network.recv‚ 3.023291e+06 disk.drbd2080.in‚0 disk.rbd1.out‚35.28857421875 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚17.156553268432617 disk.sda3.out‚ 376.02734375úË¥’é&‚(:‡ ̉äÔ´î¦ gpu.0.gpu‚12.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.834' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'¶¥H€ê&‚(:õ ̉äԇЬ  disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 network.recv‚ 3.029888e+06 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 376.7421875( proc.memory.availableMB‚ 2894.8984375 cpu‚25.339195111149834 network.sent‚ 2.5831915e+07 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125$ disk./.usageGB‚247.6193389892578# disk.dm-0.out‚53997.75048828125 disk.drbd2265.out‚0 disk.sda.out‚ 376.7421875$ memory_percent‚82.33094215393066( disk.nvme0c0n1.out‚53997.75048828125* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 376.7421875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚35.28857421875& disk.nvme0n1.out‚53997.75048828125) disk./.usagePercent‚16.61794160603316 disk.sda1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125" proc.memory.rssMB‚ 2810.9296875ü@†’ë&‚(:‡ Û‰äÔè׆ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â×H:sì&‚(:è Û‰äÔìƒÐ  disk.dm-1.out‚ 377.078125$ disk.nvme0c0n1.out‚ 54046.9296875 disk.drbd1716.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125$ memory_percent‚82.33094215393066% disk./.usageGB‚247.66057586669922 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.out‚ 377.078125( proc.memory.availableMB‚ 2894.8984375 cpu‚25.338669159781233 disk.rbd0.out‚0 disk.sda1.in‚0 network.sent‚ 2.5837929e+07 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.out‚ 54046.9296875* disk./.usagePercent‚16.620709047478297# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 network.recv‚ 3.034221e+06 disk.drbd2266.out‚0 disk.rbd1.out‚35.32080078125 disk.dm-0.out‚ 54046.9296875 disk.drbd2265.in‚0 disk.sda3.out‚ 377.078125" proc.memory.rssMB‚ 2810.9296875 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.out‚0* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚171¨ß³2í&- _wandbruntime‚8252  _runtime‚8252›e,KÈî& ' lr-AdamW/pg1‚1.2547756769110864e-05 ' lr-AdamW/pg2‚1.2547756769110864e-05 trainer/global_step‚10599 # _timestamp‚1.78841314944578e+09 _runtime‚8252.433231371 _step‚868żøñÍï&Ç ' lr-AdamW/pg2‚1.2547756769110864e-05 trainer/global_step‚10599 # _timestamp‚1.78841314944578e+09 _runtime‚8252.433231371 _step‚868 ' lr-AdamW/pg1‚1.2547756769110864e-05ä΀,U2ð&- _wandbruntime‚8252  _runtime‚8252FCÕè'ñ&âO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200033170 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8840890301246 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154023420 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624794340461056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624794340461056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚428 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140482276 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚599787522106368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚869 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8873556036608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8840235863490 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚681915684350464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615921440786944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154023422 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154023422 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615921458789888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚608627757969858 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200013132 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9307864054784 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354036552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615921458789888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140482706 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8873556036608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680!Öň#ò&çO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8840890301246 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154023420 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624794340461056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624794340461056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚428 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140482276 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚599787522106368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚869 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8873556036608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8840235863490 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚681915684350464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615921440786944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154023422 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154023422 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615921458789888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚608627757969858 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200013132 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9307864054784 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_trai†yâKen_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354036552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615921458789888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140482706 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8873556036608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200033170 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚430åjüÕ 2ó&- _wandbruntime‚8253  _runtime‚8253‡ýš«ò&ô&ìM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599842443276288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8841706736418 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8841052298454 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154037438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200031586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873719101952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154037438 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624851512927744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681978205671936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615977793825792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50622270 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154037436 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681978386931200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200051650 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354069022 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9308734638592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚8253.117920164 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873719101952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140495388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50622702 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608683478535382 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624852187306496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚615977811828736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89872686 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89872684 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140494954 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚10599 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8874375477760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599842426236928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154037436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200051650 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0–'­e÷&õ&ñM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154037438 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624851512927744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681978205671936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615977793825792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50622270 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154037436 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681978386931200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200051650 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354069022 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9308734638592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚8253.117920164 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873719101952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140495388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50622702 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608683478535382 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624852187306496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚615977811828736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89872686 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89872684 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140494954 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚10599 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8874375477760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599842426236928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154037436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200051650 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599842443276288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8841706736418 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8841052298454 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154037438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200031586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873719101952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0æöe~Ò2ö&- _wandbruntime‚8253  _runtime‚8253¤MmÉ^÷&Ø .loss/FRt3/detection_loss‚1.292188736005572 0loss/FRt1/assignment_loss‚3.1284184718132018 0loss/FRt3/assignment_loss‚1.2397368466644956 0loss/SRqqt1/assignment_loss‚1.37987417068619 2loss/SRqqt3/assignment_loss‚1.0381816168400375 1loss/SRqqt3/detection_loss‚0.9480852207676931 _runtime‚8253.119096715 /loss/FRt4/detection_loss‚2.9079562202095985 1loss/SRqqt4/detection_loss‚1.8612138964846485 .loss/FBt2/detection_loss‚3.077580189704895 /loss/FBt3/detection_loss‚0.5675995799330329 epoch‚8 _step‚871 /loss/FRt2/assignment_loss‚3.135701304528771 0loss/FRt4/assignment_loss‚2.3616875540465117 2loss/SRqqt4/assignment_loss‚1.4278762563605045 2loss/FBt4/assignment_loss‚0.002636796358274296 .loss/FRt1/detection_loss‚2.979868575334549 1loss/FBt1/assignment_loss‚0.19818960668053193 % _timestamp‚1.7884131501317124e+09 1loss/FBt2/assignment_loss‚0.20664558000862598 0loss/FBt3/assignment_loss‚0.0912012920141151 1loss/SRqqt2/detection_loss‚2.0370960793596633 /loss/FBt1/detection_loss‚2.1711061716079714 %loss/total_loss‚41.36057032399668 trainer/global_step‚10599 2loss/SRqqt2/assignment_loss‚1.7834974339707417 /loss/FRt2/detection_loss‚2.2422835640508296 1loss/SRqqt1/detection_loss‚2.3434183513506865 /loss/FBt4/detection_loss‚2.9385268092155457(™æúcø&Ý _runtime‚8253.119096715 /loss/FRt4/detection_loss‚2.9079562202095985 1loss/SRqqt4/detection_loss‚1.8612138964846485 .loss/FBt2/detection_loss‚3.077580189704895 /loss/FBt3/detection_loss‚0.5675995799330329 epoch‚8 _step‚871 /loss/FRt2/assignment_loss‚3.135701304528771 0loss/FRt4/assignment_loss‚2.3616875540465117 2loss/SRqqt4/assignment_loss‚1.4278762563605045 2loss/FBt4/assignment_loss‚0.002636796358274296 .loss/FRt1/detection_loss‚2.979868575334549 1loss/FBt1/assignment_loss‚0.19818960668053193 % _timestamp‚1.7884131501317124e+09 1loss/FBt2/assignment_loss‚0.20664558000862598 0loss/FBt3/assignment_loss‚0.0912012920141151 1loss/SRqqt2/detection_loss‚2.0370960793596633 /loss/FBt1/detection_loss‚2.1711061716079714 %loss/total_loss‚41.36057032399668 trainer/global_step‚10599 2loss/SRqqt2/assignment_loss‚1.7834974339707417 /loss/FRt2/detection_loss‚2.2422835640508296 1loss/SRqqt1/detection_loss‚2.3434183513506865 /loss/FBt4/detection_loss‚2.9385268092155457 .loss/FRt3/detection_loss‚1.292188736005572 0loss/FRt1/assignment_loss‚3.1284184718132018 0loss/FRt3/assignment_loss‚1.2397368466644956 0loss/SRqqt1/assignment_loss‚1.37987417068619 2loss/SRqqt3/assignment_loss‚1.0381816168400375 1loss/SRqqt3/detection_loss‚0.9480852207676931ç¥éÃë’ù&‚(:‡ ê‰äÔé˜ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚187.054' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.4108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØW̳rú&‚(:ç ê‰äÔúÕ  disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1953125 cpu‚25.33811915235643 proc.cpu.threads‚17 disk.sda.out‚ 377.65625 disk.rbd0.out‚0 disk.drbd1716.out‚0 network.recv‚ 3.038693e+06% disk./.usageGB‚247.68752670288086 disk.sda3.in‚ 0.1953125* proc.memory.percent‚17.156553268432617 disk.drbd2265.out‚0 disk.drbd2266.in‚0# disk.nvme0n1.out‚54113.66796875 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda1.out‚0 disk.rbd1.out‚35.32080078125 network.sent‚ 2.5843932e+07* disk./.usagePercent‚16.622517740707703 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.dm-1.out‚ 377.65625$ memory_percent‚82.33113288879395 disk.drbd2080.out‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚54113.66796875 disk.dm-0.out‚54113.66796875( proc.memory.availableMB‚ 2894.8671875" proc.memory.rssMB‚ 2810.9296875 disk.sda3.out‚ 377.65625hy®û&‚(:… ù‰äÔ¬øû gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽP? ƒü&‚(:ø ù‰äÔ þ# disk.dm-0.out‚54169.46337890625" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.sda.out‚ 378.19921875 disk.sda2.out‚0( disk.nvme0c0n1.out‚54169.46337890625 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚54169.46337890625* proc.memory.percent‚17.156553268432617 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0) proc.memory.availableMB‚ 2894.87109375$ disk./.usageGB‚247.6896629333496 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 35.3837890625 disk.dm-0.in‚ 1187.62109375$ memory_percent‚82.33110904693604 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 378.19921875 disk.sda1.out‚0 cpu‚25.337600351650437 network.sent‚ 2.595284e+07 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 378.19921875 disk.drbd2266.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 network.recv‚ 3.045619e+06* disk./.usagePercent‚16.622661104926877 disk.sda.in‚ 0.1953125¢Æ¿“2ý&- _wandbruntime‚8289  _runtime‚8289ÿš HÈþ& ' lr-AdamW/pg1‚1.1732278801630315e-05 ' lr-AdamW/pg2‚1.1732278801630315e-05 trainer/global_step‚10649 # _timestamp‚1.78841318617683e+09 _runtime‚8289.164234376 _step‚872ìZ¸­Íÿ&Ç ' lr-AdamW/pg1‚1.1732278801630315e-05 ' lr-AdamW/pg2‚1.1732278801630315e-05 trainer/global_step‚10649 # _timestamp‚1.78841318617683e+09 _runtime‚8289.164234376 _step‚872è ÇŸ­2€'- _wandbruntime‚8289  _runtime‚8289b 'âO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50846495 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602532728634368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355663404 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200959222 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚611414471314382 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154724222 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627653652574720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200939184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685041393500672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8914541842432 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154724220 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8913885479936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627652978209280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚618739110732288 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141119865 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154724222 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50846947 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200959222 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚627653652574720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚618739092729344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884131861775355e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200939184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154724220 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141119411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90272916 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8914541842432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452UF/,í'‚'çO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8881725640654 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627652978209280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9351195751424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚602532745673728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9351036321792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚675690357178880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8881071202898 _runtime‚8289.165217992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚618739110732288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚675690381118976 _step‚873 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355683444 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685041576870400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627652978209280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚618739110732288 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141119865 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154724222 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50846947 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200959222 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚627653652574720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚618739092729344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884131861775355e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200939184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154724220 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141119411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90272916 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8914541842432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50846495 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602532728634368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355663404 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200959222 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚611414471314382 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154724222 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627653652574720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200939184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685041393500672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8914541842432 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154724220 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8913885479936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627710150675968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90281086 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715940 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685103920333312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚618795445768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90281084 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200957638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627710825054720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚675752031856640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154738236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9351912416768 % _timestamp‚1.7884131868769994e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚675752007916544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611469520402790 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355715940 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚618795445768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚8289.865161371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚602587649804288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602587632764928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200977702 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200977702 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627710825054720 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8915361283584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8915361283584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154738236 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141132169 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8882542075826 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154738238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50851511 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154738238 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355695874 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844êN ñ…'ñM trainer/global_step‚10649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚618795445768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90281084 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200957638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627710825054720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚675752031856640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154738236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9351912416768 % _timestamp‚1.7884131868769994e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚675752007916544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611469520402790 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355715940 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚618795445768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚8289.865161371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚602587649804288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602587632764928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200977702 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200977702 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627710825054720 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8915361283584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8915361283584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154738236 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141132169 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.al8Ž located‚ 8882542075826 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154738238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50851511 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154738238 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355695874 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8914704907776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200957638 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚874 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627710150675968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90281086 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715940 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685103920333312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0êå)­82†'- _wandbruntime‚8289  _runtime‚8289ìèA^‡'Ø 0loss/FRt3/assignment_loss‚1.2322736258776683 0loss/FRt4/assignment_loss‚2.8045932997104734 1loss/FBt2/assignment_loss‚0.13396331698168068 1loss/SRqqt2/detection_loss‚1.6541412319495994 &loss/total_loss‚39.941251562653775 0loss/FRt1/assignment_loss‚2.7068522434968214 /loss/FRt1/detection_loss‚3.0986472849662485 1loss/SRqqt4/detection_loss‚1.8812877544689746 epoch‚8 2loss/SRqqt4/assignment_loss‚1.8134268253509487 /loss/FRt4/detection_loss‚2.2310207375260287 /loss/FBt1/detection_loss‚2.2438395420710244 /loss/FBt3/detection_loss‚0.6333814244457879 % _timestamp‚1.7884131868789437e+09 1loss/FBt3/assignment_loss‚0.09245211054698609 /loss/FRt2/assignment_loss‚3.138503921586414 1loss/SRqqt3/assignment_loss‚0.799709708133543 2loss/SRqqt1/assignment_loss‚1.2557360394192594 .loss/FRt2/detection_loss‚2.338408009963739 0loss/SRqqt3/detection_loss‚1.044087500113881 2loss/SRqqt2/assignment_loss‚1.2470693873628131 0loss/FBt1/assignment_loss‚0.3490020554396319 .loss/FBt4/detection_loss‚2.574024200439453 _step‚875 1loss/FBt4/assignment_loss‚0.18987236213999698 /loss/FRt3/detection_loss‚1.5104419743677355 1loss/SRqqt1/detection_loss‚2.5624974497726987 .loss/FBt2/detection_loss‚2.406019556522369 trainer/global_step‚10649 _runtime‚8289.866343682M®Hcˆ'Ý 1loss/FBt3/assignment_loss‚0.09245211054698609 /loss/FRt2/assignment_loss‚3.138503921586414 1loss/SRqqt3/assignment_loss‚0.799709708133543 2loss/SRqqt1/assignment_loss‚1.2557360394192594 .loss/FRt2/detection_loss‚2.338408009963739 0loss/SRqqt3/detection_loss‚1.044087500113881 2loss/SRqqt2/assignment_loss‚1.2470693873628131 0loss/FBt1/assignment_loss‚0.3490020554396319 .loss/FBt4/detection_loss‚2.574024200439453 _step‚875 1loss/FBt4/assignment_loss‚0.18987236213999698 /loss/FRt3/detection_loss‚1.5104419743677355 1loss/SRqqt1/detection_loss‚2.5624974497726987 .loss/FBt2/detection_loss‚2.406019556522369 trainer/global_step‚10649 _runtime‚8289.866343682 0loss/FRt3/assignment_loss‚1.2322736258776683 0loss/FRt4/assignment_loss‚2.8045932997104734 1loss/FBt2/assignment_loss‚0.13396331698168068 1loss/SRqqt2/detection_loss‚1.6541412319495994 &loss/total_loss‚39.941251562653775 0loss/FRt1/assignment_loss‚2.7068522434968214 /loss/FRt1/detection_loss‚3.0986472849662485 1loss/SRqqt4/detection_loss‚1.8812877544689746 epoch‚8 2loss/SRqqt4/assignment_loss‚1.8134268253509487 /loss/FRt4/detection_loss‚2.2310207375260287 /loss/FBt1/detection_loss‚2.2438395420710244 /loss/FBt3/detection_loss‚0.6333814244457879 % _timestamp‚1.7884131868789437e+09ë¦ÝÈH’‰'‚(:‡ ˆŠäÔï“« gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íJ™€Š'‚(:õ ˆŠäÔ“§º  disk.dm-1.in‚ 0.1953125% disk.nvme0n1.out‚54220.9541015625* disk./.usagePercent‚16.624010008624793 disk.drbd2080.out‚0# disk.nvme0c0n1.in‚ 1187.62109375' disk.nvme0c0n1.out‚54220.9541015625! disk.nvme0n1.in‚ 1187.62109375 proc.cpu.threads‚17 disk.drbd2265.out‚0# memory_percent‚82.3314905166626) proc.memory.availableMB‚ 2894.80859375$ disk./.usageGB‚247.7097625732422 disk.drbd2080.in‚0 network.recv‚ 3.050092e+06 disk.drbd2265.in‚0 disk.sda.out‚ 378.86328125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 2.5958367e+07 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.dm-0.in‚ 1187.62109375" disk.dm-0.out‚54220.9541015625 cpu‚25.337108899952295 disk.dm-1.out‚ 378.86328125 disk.sda3.out‚ 378.86328125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 35.3837890625" proc.memory.rssMB‚ 2810.9296875* proc.memory.percent‚17.156553268432617 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0ÅÈ.‰‹'‚(:’ —ŠäÔšìž gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.296' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.459199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜¥eb|Œ'‚(:ñ —ŠäÔ’У  network.sent‚ 2.6066987e+07 disk.drbd1716.in‚0$ memory_percent‚82.33151435852051 cpu‚25.336563334005305% disk./.usageGB‚247.72784042358398 disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.156553268432617! disk.nvme0n1.in‚ 1187.62109375# disk.dm-0.out‚54273.49462890625 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.rbd1.out‚35.44677734375 disk.dm-1.out‚ 379.09375 disk.sda.out‚ 379.09375 disk.sda3.out‚ 379.09375 disk.drbd2080.in‚0* disk./.usagePercent‚16.625223228329556& disk.nvme0n1.out‚54273.49462890625 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1187.62109375 network.recv‚ 3.056419e+06( disk.nvme0c0n1.out‚54273.49462890625 disk.drbd1716.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 2894.8046875" proc.memory.rssMB‚ 2810.9296875# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0ÁëF'‚(:’ ¦ŠäÔø°õ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.510799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÐÖä„Ž'‚(:ù ¦ŠäÔ£  disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0( proc.memory.availableMB‚ 2894.5546875 disk.rbd1.out‚35.44677734375 disk.sda.out‚ 379.62109375 disk.sda2.in‚0( disk.nvme0c0n1.out‚54322.20556640625! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚54322.20556640625 disk.dm-0.in‚ 1187.62109375 disk.sda1.out‚0" proc.memory.rssMB‚ 2810.9296875 cpu‚25.336049648082938 proc.cpu.threads‚17 network.recv‚ 3.060676e+06# disk.dm-0.out‚54322.20556640625 disk.drbd2080.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 disk.sda1.in‚0) disk./.usagePercent‚16.62679025044664 disk.sda3.out‚ 379.62109375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0$ memory_percent‚82.33304023742676* proc.memory.percent‚17.156553268432617% disk./.usageGB‚247.75119018554688 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 379.62109375 disk.sda2.out‚0 network.sent‚ 2.6072888e+07 disk.drbd2265.in‚0.è×2'- _wandbruntime‚8326  _runtime‚8326uÌ6Ê'Ä ' lr-AdamW/pg2‚1.0943016791689062e-05 trainer/global_step‚10699 % _timestamp‚1.7884132232466009e+09 _runtime‚8326.234036588 _step‚876 ' lr-AdamW/pg1‚1.0943016791689062e-05“5F,Ï‘'É % _timestamp‚1.7884132232466009e+09 _runtime‚8326.234036588 _step‚876 ' lr-AdamW/pg1‚1.0943016791689062e-05 ' lr-AdamW/pg2‚1.0943016791689062e-05 trainer/global_step‚10699ìÑÀò12’'- _wandbruntime‚8326  _runtime‚8326/R1è'“'âO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201885274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8954871285760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630511615957504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8955527648256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51074886 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8954871285760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚877 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357310296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51074450 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9394305684992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201885274 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚621556744671744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621556762674688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621556744671744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8922560980062 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201865236 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155425022 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚614200513181790 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8955527648256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621556762674688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614199841704674 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155425020 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚630512290322944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚630512290322944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201865236 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414¸üýÁí'”'çO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155425022 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155425020 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884132232472944e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630511615957504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8955527648256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51074886 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8954871285760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚877 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357310296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51074450 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9394305684992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201885274 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚621556744671744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621556762674688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621556744671744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8922560980062 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201865236 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155425022 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚614200513181790 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8955527648256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621556762674688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614199841704674 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155425020 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚630512290322944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚630512290322944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201865236 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201885274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8954871285760 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141769008 REDeviceStatsMonitor.ô ¸ on_train_batch_end/inactive_split.small_pool.freed‚51079087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚614255562270198 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚630569462802944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚878 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8922722977270 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357342792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8955690713600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155439038 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚678834544799744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚621613115713536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90689484 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51079522 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141768571 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚614256233747522 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚10699 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8923377415234 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201883690 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8956347089408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884132239379663e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚621613097710592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155439036 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688229731057664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚678834568739840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9395162317824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322726 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201903754 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚630568788424192 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0TG¿Mö&—'ðM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8922722977270 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357342792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8955690713600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155439038 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚678834544799744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚621613115713536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90689484 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51079522 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141768571 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚614256233747522 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚10699 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8923377415234 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201883690 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8956347089408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884132239379663e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚621613097710592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155439036 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688229731057664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚678834568739840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9395162317824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322726 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201903754 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚630568788424192 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚630569462802944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357322726 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201883690 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201903754 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8955690713600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155439036 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9394998707200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚605332856332288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141769008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51079087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚614255562270198 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚630569462802944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚878 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1î¨õÔU2˜'- _wandbruntime‚8326  _runtime‚8326ÞÃ`i[™'Õ /loss/FRt4/assignment_loss‚2.598568114828556 1loss/FBt4/assignment_loss‚0.16980850254185498 0loss/SRqqt2/detection_loss‚2.415690072908484 -loss/FBt2/detection_loss‚2.60187406539917 2loss/SRqqt1/assignment_loss‚1.6376717362414908 2loss/SRqqt3/assignment_loss‚0.9463821980276738 1loss/FBt3/assignment_loss‚0.03558527211348197 /loss/FRt1/detection_loss‚2.9962196477821896 trainer/global_step‚10699 _step‚879 2loss/SRqqt2/assignment_loss‚1.5173409278022831 1loss/FBt1/assignment_loss‚0.14259541006426094 1loss/SRqqt3/detection_loss‚1.1237350908870047 /loss/FBt3/detection_loss‚0.6219930171966552 _runtime‚8326.927345113 2loss/SRqqt4/assignment_loss‚1.1457267536623579 /loss/FRt3/detection_loss‚1.6419685387655216 1loss/SRqqt4/detection_loss‚1.5540254616512443 %loss/total_loss‚41.06884733858078 /loss/FRt2/assignment_loss‚2.808990894092454 0loss/FRt3/assignment_loss‚0.8884853218116012 epoch‚8 % _timestamp‚1.7884132239399347e+09 .loss/FRt2/detection_loss‚2.452894844197411 0loss/SRqqt1/detection_loss‚2.574830678878007 /loss/FBt1/detection_loss‚2.1161063058035716 /loss/FRt4/detection_loss‚2.1412301171333232 /loss/FRt1/assignment_loss‚2.858122704284532 0loss/FBt2/assignment_loss‚0.2066252801567316 /loss/FBt4/detection_loss‚3.8723763823509216÷ ª`š'Ú /loss/FRt1/assignment_loss‚2.858122704284532 0loss/FBt2/assignment_loss‚0.2066252801567316 /loss/FBt4/detection_loss‚3.8723763823509216 /loss/FRt4/assignment_loss‚2.598568114828556 1loss/FBt4/assignment_loss‚0.16980850254185498 0loss/SRqqt2/detection_loss‚2.415690072908484 -loss/FBt2/detection_loss‚2.60187406539917 2loss/SRqqt1/assignment_loss‚1.6376717362414908 2loss/SRqqt3/assignment_loss‚0.9463821980276738 1loss/FBt3/assignment_loss‚0.03558527211348197 /loss/FRt1/detection_loss‚2.9962196477821896 trainer/global_step‚10699 _step‚879 2loss/SRqqt2/assignment_loss‚1.5173409278022831 1loss/FBt1/assignment_loss‚0.14259541006426094 1loss/SRqqt3/detection_loss‚1.1237350908870047 /loss/FBt3/detection_loss‚0.6219930171966552 _runtime‚8326.927345113 2loss/SRqqt4/assignment_loss‚1.1457267536623579 /loss/FRt3/detection_loss‚1.6419685387655216 1loss/SRqqt4/detection_loss‚1.5540254616512443 %loss/total_loss‚41.06884733858078 /loss/FRt2/assignment_loss‚2.808990894092454 0loss/FRt3/assignment_loss‚0.8884853218116012 epoch‚8 % _timestamp‚1.7884132239399347e+09 .loss/FRt2/detection_loss‚2.452894844197411 0loss/SRqqt1/detection_loss‚2.574830678878007 /loss/FBt1/detection_loss‚2.1161063058035716 /loss/FRt4/detection_loss‚2.1412301171333232ïD‰æÚ›'‚(:’ µŠäÔ¾„§ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.011599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô˜Ýisœ'‚(:è µŠäÔõ»  network.sent‚ 2.6079208e+07 disk.sda3.out‚ 380.421875 disk.rbd0.in‚0% disk.nvme0c0n1.out‚54369.18359375" proc.memory.rssMB‚ 2810.9296875 disk.drbd2266.in‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0* proc.memory.percent‚17.156553268432617 disk.rbd0.out‚0 network.recv‚ 3.064965e+06* disk./.usagePercent‚16.627487102954845 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚35.44677734375 disk.drbd2265.out‚0 cpu‚25.33553444666701 disk.sda.out‚ 380.421875 disk.drbd2266.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1187.62109375 proc.cpu.threads‚17$ disk./.usageGB‚247.7615737915039 disk.sda.in‚ 0.1953125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.dm-1.out‚ 380.421875' proc.memory.availableMB‚ 2894.765625 disk.sda2.out‚0 disk.dm-0.out‚54369.18359375 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125$ memory_percent‚82.33175277709961 disk.drbd2080.in‚0 disk.drbd2080.out‚0# disk.nvme0n1.out‚54369.18359375¶’Vê'‚(:’ ÄŠäÔ¸«¢ gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.611' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.322200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸3,zž'‚(:ï ÄŠäÔŽ˜…  disk.sda3.out‚ 381.02734375) disk./.usagePercent‚16.62879632548495 disk.dm-1.out‚ 381.02734375 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1187.62109375# disk.nvme0n1.out‚54412.41015625) proc.memory.availableMB‚ 2894.76953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda2.in‚0% disk.nvme0c0n1.out‚54412.41015625 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375 network.sent‚ 2.6187266e+07 disk.drbd2265.out‚0 proc.cpu.threads‚17 network.recv‚ 3.071059e+06 disk.drbd2265.in‚0 disk.sda.out‚ 381.02734375# disk.nvme0c0n1.in‚ 1187.62109375# memory_percent‚82.3317289352417 cpu‚25.335026021555475 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2810.9296875 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.rbd1.in‚0 disk.dm-0.out‚54412.41015625$ disk./.usageGB‚247.7810821533203 disk.dm-1.in‚ 0.1953125 disk.sda1.out‚0 disk.rbd1.out‚35.44677734375* proc.memory.percent‚17.156553268432617|ÑËb2Ÿ'- _wandbruntime‚8363  _runtime‚8363 Œ 8È ' & lr-AdamW/pg1‚1.018013522030701e-05 & lr-AdamW/pg2‚1.018013522030701e-05 trainer/global_step‚10749 % _timestamp‚1.7884132601149867e+09 _runtime‚8363.102404127 _step‚880G`ÃÍ¡'Ç _step‚880 & lr-AdamW/pg1‚1.018013522030701e-05 & lr-AdamW/pg2‚1.018013522030701e-05 trainer/global_step‚10749 % _timestamp‚1.7884132601149867e+09 _runtime‚8363.102404127ð8G6n2¢'- _wandbruntime‚8363  _runtime‚8363‹m;Òã£'áO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202791288 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156125822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91089718 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚681855430945280 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91089716 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358937148 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142392701 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358917108 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202791288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51303438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚681855454885376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10749 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚633370253705728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633370928071168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8963396319470 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624374414617088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142393156 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9437255826432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9437083813888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚608023141690368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788413260115662e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156125820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202811326 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624374396614144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8962741881714 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8995857091584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202811326 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚608023158729728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633370928071168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691292514759168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616986555049198 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156125820 ZLDeviceStatsMonitor.on_train_bfý~ atch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚8363.103308263 _step‚881 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633370253705728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156125822 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚624374396614144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8996513454080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616985883572082 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚624374414617088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8995857091584 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633370928071168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691292514759168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616986555049198 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156125820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚8363.103308263 _step‚881 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633370253705728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156125822 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚624374396614144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8996513454080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616985883572082 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚624374414617088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8995857091584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51303438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚681855454885376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10749ñÆ\t2¥'- _wandbruntime‚8363  _runtime‚8363T?Æò&¦'ìM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9438111377920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8996676519424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚691355217000960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚681917081682944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚617042275614930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358949578 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156139836 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202809742 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884132608167987e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8996676519424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚624430749652992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚691355031547392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8997332895232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51308027 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202809742 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚681917105623040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9437949864448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚608078062860288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969644 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚617041604137606 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142405472 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358969644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚624430767655936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚608078045820928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949578 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156139838 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156139836 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202829806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91097886 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202829806 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633428100551168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10749 _step‚882 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚624430767655936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142405913>%ÈÔ÷&§'ñM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633427426172416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156139836 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202829806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91097886 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202829806 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633428100551168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10749 _step‚882 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚624430767655936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142405913 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9438111377920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8996676519424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚691355217000960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚681917081682944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚617042275614930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358949578 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156139836 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202809742 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884132608167987e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8996676519424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚624430749652992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚691355031547392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8997332895232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51308027 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202809742 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚681917105623040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9437949864448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚608078062860288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969644 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚617041604137606 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142405472 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358969644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚624430767655936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚608078045820928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949578 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156139838 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636228891453952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627192048556544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360564000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9004231658878 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203717340 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚619771925439490 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚636229565819392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480254662656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203737378 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143029530 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684937991768576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9037499259904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚610768348218368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91498118 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203737378 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚10799 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156826622 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156826620 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9036842897408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9003577221122 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚636228891453952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143029076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51530960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9037499259904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684937967828480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627192048556544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636228891453952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627192048556544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360564000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9004231658878 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203717340 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚619771925439490 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚636229565819392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480254662656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203737378 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143029530 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684937991768576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9037499259904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚610768348218368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91498118 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203737378 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚10799 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156826622 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156826620 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9036842897408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9003577221122õüŸÍ2·'- _wandbruntime‚8401  _runtime‚8401³x<–N¸'ìM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203755858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51535545 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627248401595392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203755858 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚10799 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360576430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143041829 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9481129558528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884132980697763e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚636286738299392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156840636 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 _runtime‚8401.058057594 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9480976433664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9004393656086 UGDeviceStatsMonitor.on_train_batch_end/rÚ¯}¤eserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚636286738299392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627248401595392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156840636 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684999618566144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156840638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚610823252348928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9038318701056 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203735794 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9005048094050 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627248419598336 _step‚886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9037662325248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143042284 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143041829 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9481129558528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884132980697763e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚636286738299392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156840636 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 _runtime‚8401.058057594 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9480976433664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9004393656086 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚636286738299392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627248401595392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156840636 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684999618566144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156840638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚610823252348928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9038318701056 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203735794 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9005048094050 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627248419598336 _step‚886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9037662325248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143042284 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613513554746368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9045066998286 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697544031504384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639087529202176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630009700498944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚622557967306898 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362190852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51760714 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9523502852608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630009718501888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚613513571785728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157527422 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9077828703232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630009700498944 JOÇ%_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚622557967306898 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362190852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51760714 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9523502852608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630009718501888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚613513571785728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157527422 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9077828703232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630009700498944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613513554746368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9045066998286 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697544031504384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639087529202176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ù}a2Ç'- _wandbruntime‚8438  _runtime‚8438E*±Ôñ&È'ëM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚890 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362203282 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639145376047616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51764791 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157541438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9079304506880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204681910 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613568475916288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204661846 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362223348 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚697606541242368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9045228995494 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362223348 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚688082155449344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91914684 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697606347400192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143679475 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788413335139882e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639145376047616 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143679919 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91914686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51765233 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚622614359349746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚10849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157541436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639144701668864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9078648131072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362203282 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630066071540736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9078648131072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844mƒÕö&É'ðM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚890 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362203282 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639145376047616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51764791 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157541438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9079304506880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204681910 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613568475916288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204661846 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362223348 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚697606541242368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9045228995494 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362223348 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚688082155449344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91914684 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697606347400192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143679475 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788413335139882e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639145376047616 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143679919 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91914686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51765233 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚622614359349746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚10849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157541436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639144701668864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9078648131072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362203282 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630066071540736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9078648131072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844úÿöÑ2Ê'- _wandbruntime‚8438  _runtime‚8438å¾7ÕZË'Ô /loss/FRt4/assignment_loss‚2.481280341744423 2loss/SRqqt3/assignment_loss‚0.8664119882493347 1loss/SRqqt3/detection_loss‚0.8565158707917564 _step‚891 0loss/FRt1/assignment_loss‚3.0307639922414507 % _timestamp‚1.7884133351416025e+09 1loss/FBt3/assignment_loss‚0.10738196650950371 /loss/FBt3/detection_loss‚0.5915088316251058 _runtime‚8438.128999286 0loss/FRt2/assignment_loss‚2.8079934088807357 0loss/FBt1/assignment_loss‚0.2655871534370817 1loss/FBt4/assignment_loss‚0.08442606809603603 /loss/FRt2/detection_loss‚2.5457219043256423 /loss/FRt3/detection_loss‚1.3997612426678339 /loss/FRt4/detection_loss‚2.6004073195159436 0loss/SRqqt4/detection_loss‚1.679259404258908 0loss/FRt3/assignment_loss‚1.2224220395088197 0loss/SRqqt2/assignment_loss‚1.39711438886383 2loss/SRqqt4/assignment_loss‚1.1322016555042762 .loss/FBt1/detection_loss‚2.455877935886383 .loss/FBt2/detection_loss‚2.218685040871302 %loss/total_loss‚39.42687494272886 epoch‚8 .loss/FBt4/detection_loss‚2.416523790359497 2loss/SRqqt1/assignment_loss‚1.8694293063420515 /loss/FRt1/detection_loss‚3.0628841732229506 /loss/SRqqt1/detection_loss‚2.45037530706479 1loss/SRqqt2/detection_loss‚1.6595284239042039 trainer/global_step‚10849 1loss/FBt2/assignment_loss‚0.22481338885700097_âÌ«_Ì'Ù 1loss/FBt3/assignment_loss‚0.10738196650950371 /loss/FBt3/detection_loss‚0.5915088316251058 _runtime‚8438.128999286 0loss/FRt2/assignment_loss‚2.8079934088807357 0loss/FBt1/assignment_loss‚0.2655871534370817 1loss/FBt4/assignment_loss‚0.08442606809603603 /loss/FRt2/detection_loss‚2.5457219043256423 /loss/FRt3/detection_loss‚1.3997612426678339 /loss/FRt4/detection_loss‚2.6004073195159436 0loss/SRqqt4/detection_loss‚1.679259404258908 0loss/FRt3/assignment_loss‚1.2224220395088197 0loss/SRqqt2/assignment_loss‚1.39711438886383 2loss/SRqqt4/assignment_loss‚1.1322016555042762 .loss/FBt1/detection_loss‚2.455877935886383 .loss/FBt2/detection_loss‚2.218685040871302 %loss/total_loss‚39.42687494272886 epoch‚8 .loss/FBt4/detection_loss‚2.416523790359497 2loss/SRqqt1/assignment_loss‚1.8694293063420515 /loss/FRt1/detection_loss‚3.0628841732229506 /loss/SRqqt1/detection_loss‚2.45037530706479 1loss/SRqqt2/detection_loss‚1.6595284239042039 trainer/global_step‚10849 1loss/FBt2/assignment_loss‚0.22481338885700097 /loss/FRt4/assignment_loss‚2.481280341744423 2loss/SRqqt3/assignment_loss‚0.8664119882493347 1loss/SRqqt3/detection_loss‚0.8565158707917564 _step‚891 0loss/FRt1/assignment_loss‚3.0307639922414507 % _timestamp‚1.7884133351416025e+09ûáxbýÍ'‚(:‡ ž‹äÔñ´Ú gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu0ZÕ.0.powerPercent‚24.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ù9€Î'‚(:õ ž‹äÔ«û†  network.recv‚ 3.101471e+06 disk.dm-0.in‚ 1187.62109375# disk.nvme0c0n1.in‚ 1187.62109375 cpu‚25.33197862404974 disk.sda2.out‚0) disk./.usagePercent‚16.63447354856426! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.drbd1716.in‚0* proc.memory.percent‚17.156553268432617 disk.rbd1.in‚0" disk.dm-0.out‚54694.4130859375" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.rbd1.out‚35.59521484375 disk.dm-1.in‚ 0.1953125 disk.rbd0.in‚0 disk.drbd1716.out‚0$ memory_percent‚82.33325481414795 network.sent‚ 2.6427306e+07$ disk./.usageGB‚247.8656768798828 disk.sda.out‚ 384.05078125 disk.sda3.out‚ 384.05078125 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚54694.4130859375) proc.memory.availableMB‚ 2894.51953125 disk.dm-1.out‚ 384.05078125 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0% disk.nvme0n1.out‚54694.4130859375 disk.sda.in‚ 0.1953125NãöÂÏ'‚(:’ ­‹äÔ‚±† gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.595800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&•<ŽÐ'‚(:ô ­‹äÔ’ù“ " disk.dm-0.out‚54744.2685546875 disk.sda.out‚ 384.6796875 disk.sda2.out‚0 network.sent‚ 2.6536538e+07 disk.dm-1.out‚ 384.6796875 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.sda1.in‚0) proc.memory.availableMB‚ 2894.48828125 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚35.62744140625% disk.nvme0n1.out‚54744.2685546875 disk.dm-0.in‚ 1187.62109375 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚82.33344554901123" proc.memory.rssMB‚ 2810.9296875 cpu‚25.33149817126281 disk.drbd2266.out‚0* disk./.usagePercent‚16.635734641677928% disk./.usageGB‚247.88446807861328 disk.drbd2265.out‚0 disk.sda3.out‚ 384.6796875* proc.memory.percent‚17.156553268432617' disk.nvme0c0n1.out‚54744.2685546875 disk.drbd1716.in‚0 disk.drbd2266.in‚0 network.recv‚ 3.108863e+06 disk.sda3.in‚ 0.1953125 proc.cpu.threads‚17Î2,Ò‘Ñ'‚(:† ¼‹äÔ““ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q•OxmÒ'‚(:â ¼‹äÔƽ  disk.rbd1.in‚0 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda3.out‚385.375$ memory_percent‚82.33344554901123 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.156553268432617 disk.sda2.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚385.375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚35.62744140625 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.dm-0.out‚54794.83984375 disk.drbd2266.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2810.9296875 network.recv‚ 3.112455e+06% disk./.usageGB‚247.85900497436523 disk.sda.out‚385.375 disk.rbd0.in‚0) proc.memory.availableMB‚ 2894.48828125 disk.sda3.in‚ 0.1953125 network.sent‚ 2.6541825e+07# disk.nvme0n1.out‚54794.83984375 cpu‚25.33909052883257* disk./.usagePercent‚16.634025791386875% disk.nvme0c0n1.out‚54794.83984375 disk.drbd1716.out‚0ƒz"?’Ó'‚(:‡ Ë‹äÔˆø gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚184.412' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0· JŒ„Ô'‚(:ù Ë‹äÔµ©¸ $ memory_percent‚81.94606304168701 proc.cpu.threads‚17) disk./.usagePercent‚16.63586392548272! disk.nvme0n1.in‚ 1187.62109375 network.sent‚ 2.6547969e+07( disk.nvme0c0n1.out‚54852.42822265625 disk.drbd1716.out‚0 disk.sda3.out‚ 385.87890625 disk.drbd2266.in‚0# disk.dm-0.out‚54852.42822265625 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.1953125 network.recv‚ 3.116883e+06% disk./.usageGB‚247.88639450073242 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2810.9296875# disk.nvme0c0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 385.87890625 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.156553268432617 cpu‚25.347086813883315 disk.drbd1716.in‚0 disk.sda.out‚ 385.87890625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 35.6904296875 disk.drbd2080.in‚0) proc.memory.availableMB‚ 2957.95703125& disk.nvme0n1.out‚54852.42822265625 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0†>ÔáÕ'‚(:„ Ú‹äԲʖ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.816 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î÷+Ý}Ö'‚(:ò Ú‹äÔþ­   disk.dm-1.out‚ 386.359375 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 386.359375# disk.nvme0c0n1.in‚ 1187.62109375) disk./.usagePercent‚16.63612633320532 disk.rbd0.in‚0( disk.nvme0c0n1.out‚54887.89306640625! disk.nvme0n1.in‚ 1187.62109375" proc.memory.rssMB‚ 2810.9296875 disk.sda.out‚ 386.359375 disk.drbd2080.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 35.6904296875$ memory_percent‚81.94754123687744) proc.memory.availableMB‚ 2957.71484375 network.sent‚ 2.6553788e+07 network.recv‚ 3.121334e+06 disk.dm-0.in‚ 1187.62109375 disk.drbd2266.in‚0$ disk./.usageGB‚247.8903045654297 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚25.355276226001102 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚54887.89306640625# disk.dm-0.out‚54887.89306640625 ›­2×'- _wandbruntime‚8509  _runtime‚8509.|„tO*Ø'ÉT VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 158505426 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 205612497 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚642740306068480 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚616999881139200 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚10880 % _timestamp‚1.7884134062542474e+09 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 158505428 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚157 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚642740306068480 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 364101946 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 205612497 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚616999898178560 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9122126082048 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 364101946 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚633617669153280 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚701418265369088 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚626088421993426 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 158505428 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9122126082048 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 144612903 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚92486590 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚52126158 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚8509.242090319 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 158505426 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 364117925 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 205596520 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚155 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚701418402555904 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚633617687156224 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9569026110464 _step‚892 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚52126313 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚642739795235328 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚92486588 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9569139357184 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 113246720 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 137186816 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚155 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚701418402555904 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚633617687156224 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9569026110464 _step‚892 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚52126313 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚642739795235328 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚92486588 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9569139357184 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 113246720 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 137186816 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚52130661 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 205603822 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚642776990390784 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚617035174898688 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚633654499182080 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9122491208704 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚691887138932224 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚691887114992128 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚642777501223936 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884134072038023e+09 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 158516234 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚92493092 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚701456643416064y^)Ü'ØR LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884134072038023e+09 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 158516234 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚92493092 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚701456643416064 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚617035191938048 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 364136033 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚92493094 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 158516232 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9122491208704 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚157 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9569504483840 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9122984038912 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 205603822 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 364136033 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚626124587316074 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10880 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9088904463634 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚642777501223936 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚633654517185024 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9569391237120 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚633654517185024 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚626124079362322 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚701456506229248 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 364120054 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚52130661 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 205603822 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚642776990390784 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚617035174898688 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚633654499182080 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9122491208704 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚691887138932224 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚691887114992128 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚642777501223936ýžÑœÝ'‚(:‘ é‹äÔÜã™ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.229' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.245800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vF™¯~Þ'‚(:ó é‹äÔž­ " disk.dm-0.out‚54926.8447265625 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 386.7109375 disk.drbd2080.out‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2810.9296875 disk.sda.out‚ 386.7109375 disk.sda2.in‚0 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.sda3.in‚ 0.1953125 network.sent‚ 2.6560038e+07( disk./.usagePercent‚16.6373733459046# disk.nvme0c0n1.in‚ 1187.62109375' disk.nvme0c0n1.out‚54926.8447265625 disk.drbd2265.in‚0 cpu‚25.363019408240586% disk./.usageGB‚247.90888595581055 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.rbd0.in‚0 network.recv‚ 3.125567e+06 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 386.7109375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚35.75341796875% disk.nvme0n1.out‚54926.8447265625 disk.drbd2265.out‚0 disk.drbd2080.in‚0$ memory_percent‚81.94925785064697) proc.memory.availableMB‚ 2957.43359375* proc.memory.percent‚17.156553268432617 disk.dm-0.in‚ 1187.62109375 disk.drbd2266.in‚0 disk.sda1.in‚0ÆMNm2ß'- _wandbruntime‚8523  _runtime‚85239êx4à'´§ 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 particle/accuracy_6_of_6‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.44034600257873535 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4096725881099701 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.563218355178833 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4673842489719391 *jet/accuracy_2_of_4‚0.3684903383255005 particle/accuracy_3_of_4‚1 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.289814829826355 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.027300916612148285 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7398809194564819 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5794482827186584 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2636807858943939 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026729561388492584 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44951316714286804 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2085375189781189 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5604166984558105 particle/accuracy_2_of_3‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6501566171646118 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 -jet/accuracy_5_of_5‚0.0026881720405071974 *particle/specificity‚0.951553225517273 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34077510237693787 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1855158656835556 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15977101027965546 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4528508484363556 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16329120099544525 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1450265794992447 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5526841282844543 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4488506019115448 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3998498022556305 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11884669214487076 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5228611826896667 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3961279094219208 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 +jet/accuracy_1_of_1‚0.47304609417915344 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016512228175997734 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16426487267017365 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6140351295471191 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20720787346363068 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2916666567325592 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 )loss/val_total_loss‚41.14814647095561 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 +jet/accuracy_2_of_6‚0.43589746952056885 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48961666226387024 ,jet/accuracy_4_of_4‚0.005641418509185314 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8569929003715515 particle/accuracy_2_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33541956543922424 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11781811714172363 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_3_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7575757503509521 +jet/accuracy_2_of_3‚0.24746295809745789 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_5_of_6‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42596447467803955 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06226581335067749 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1427823007106781 particle/accuracy_1_of_1‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2916666567325592 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21968764066696167 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20552532374858856 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.43984678387641907 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5151978731155396 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_6‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0242468249052763 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03440660238265991 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_3_of_6‚0.1538461595773697 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 jet/accuracy_2_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5102682113647461 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03977314010262489 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5284140110015869 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.45861175656318665 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4216611087322235 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45167791843414307 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_4_of_4‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4617496728897095 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019198792055249214 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4673842489719391 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20942345261573792 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919359683990479 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10210384428501129 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4923475682735443 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.607291579246521 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.025193799287080765 particle/accuracy_5_of_5‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979721903800964 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2736004590988159 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012850729748606682 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6974282264709473 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34347444772720337 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5273220539093018 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚‡, ï©_0.9642857313156128 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16591784358024597 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13607747852802277 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909806251525879 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2460939884185791 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1150793582201004 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1633979231119156 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45448458194732666 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12586207687854767 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17831240594387054 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15029282867908478 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.040028978139162064 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 )jet/accuracy_2_of_5‚0.433563768863678 particle/accuracy_1_of_3‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3915032148361206 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5728175044059753 +jet/accuracy_2_of_2‚0.11052113026380539 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9930272102355957 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1788882315158844 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2768493890762329 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09880952537059784 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868131875991821 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_6‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3866012394428253 particle/accuracy_1_of_7‚1 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17132151126861572 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4948727786540985 particle/accuracy_1_of_2‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27196839451789856 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841078519821167 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 epoch‚8 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17730247974395752 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 particle/accuracy_4_of_6‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34320586919784546 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3890117108821869 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 (particle/f_score‚0.41056960821151733 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13111111521720886 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0258620698004961 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5865439772605896 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_6‚0.057692307978868484 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886235594749451 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23873873054981232 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 jet/accuracy_7_of_7‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986807703971863 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9920656085014343 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887440204620361 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4467380940914154 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.026497548446059227 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7884134200458431e+09 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992607593536377 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19487278163433075 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 jet/accuracy_6_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9222382307052612 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4800635576248169 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5148399472236633 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918556809425354 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08164386451244354 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19284141063690186 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.504962146282196 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42683130502700806 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34756943583488464 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46594178676605225 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3739490211009979 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6198174953460693 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005851203575730324 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.224146768450737 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909061789512634 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25831979513168335 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21005746722221375 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5308641791343689 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9912964105606079 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15977416932582855 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16079303622245789 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283686965703964 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5660421252250671 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24879783391952515 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9525601863861084 (particle/accuracy‚0.8538878560066223 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6043498516082764 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 jet/accuracy_3_of_7‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 trainer/global_step‚10880 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.990084707736969 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18993240594863892 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2916950583457947 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9928615689277649 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 _runtime‚8523.035427375 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.343494713306427 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.32575759291648865 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.36666667461395264 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5655643939971924 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39054346084594727 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2639749050140381 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 particle/accuracy_3_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15317687392234802 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4791666567325592 jet/accuracy_5_of_6‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32976195216178894 jet/accuracy_5_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39534884691238403 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5819181203842163 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21934422850608826 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5337623953819275 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17408035695552826 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4860546588897705 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4192913770675659 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.003250189358368516 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5922619104385376 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02223127707839012 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.018458781763911247 particle/accuracy_2_of_2‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6874999403953552 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17612384259700775 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4534763991832733 5 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.98167884349823 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4892559349536896 +jet/accuracy_3_of_4‚0.07809457927942276 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21450616419315338 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5596719980239868 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21843433380126953 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.42897629737854004 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5082337260246277 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47100701928138733 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18099313974380493 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2470134198665619 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.49231022596359253 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6874999403953552 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5318522453308105 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.055000703781843185 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4717042148113251 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5006289482116699 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575757503509521 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.08842805027961731 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.37777769565582275 particle/accuracy_2_of_7‚1 particle/accuracy_6_of_7‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02850877307355404 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 _step‚894 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319833755493 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4953373372554779 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5443077087402344 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02219662256538868 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013609879650175571 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46748360991477966 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.43820664286613464 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1924210637807846 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.42193594574928284 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5073891282081604 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14646103978157043 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11929824948310852 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22221305966377258 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15102343261241913 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14323411881923676 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 6"Purity/0FRt0SRqqt*FBt/event_purity‚0.8153036236763 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12260536849498749 particle/accuracy_1_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 jet/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4976148307323456 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +particle/sensitivity‚0.3261105716228485 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17730247974395752 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0¢ºI á'¹§ ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 epoch‚8 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17730247974395752 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 particle/accuracy_4_of_6‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34320586919784546 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3890117108821869 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 (particle/f_score‚0.41056960821151733 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13111111521720886 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0258620698004961 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5865439772605896 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_6‚0.057692307978868484 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886235594749451 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23873873054981232 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 jet/accuracy_7_of_7‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986807703971863 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9920656085014343 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887440204620361 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4467380940914154 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.026497548446059227 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7884134200458431e+09 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992607593536377 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19487278163433075 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 jet/accuracy_6_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9222382307052612 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4800635576248169 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5148399472236633 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918556809425354 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08164386451244354 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19284141063690186 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.504962146282196 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42683130502700806 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34756943583488464 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46594178676605225 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3739490211009979 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6198174953460693 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005851203575730324 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.224146768450737 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909061789512634 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25831979513168335 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21005746722221375 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5308641791343689 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9912964105606079 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15977416932582855 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16079303622245789 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283686965703964 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5660421252250671 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24879783391952515 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9525601863861084 (particle/accuracy‚0.8538878560066223 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6043498516082764 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 jet/accuracy_3_of_7‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 trainer/global_step‚10880 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.990084707736969 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18993240594863892 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt[ß;[ws*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2916950583457947 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9928615689277649 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 _runtime‚8523.035427375 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.343494713306427 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.32575759291648865 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.36666667461395264 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5655643939971924 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39054346084594727 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2639749050140381 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 particle/accuracy_3_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15317687392234802 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4791666567325592 jet/accuracy_5_of_6‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32976195216178894 jet/accuracy_5_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39534884691238403 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5819181203842163 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21934422850608826 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5337623953819275 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17408035695552826 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4860546588897705 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4192913770675659 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.003250189358368516 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5922619104385376 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02223127707839012 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.018458781763911247 particle/accuracy_2_of_2‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6874999403953552 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17612384259700775 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4534763991832733 5 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.98167884349823 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4892559349536896 +jet/accuracy_3_of_4‚0.07809457927942276 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21450616419315338 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5596719980239868 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21843433380126953 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.42897629737854004 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5082337260246277 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47100701928138733 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18099313974380493 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2470134198665619 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.49231022596359253 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6874999403953552 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5318522453308105 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.055000703781843185 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4717042148113251 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5006289482116699 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575757503509521 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.08842805027961731 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.37777769565582275 particle/accuracy_2_of_7‚1 particle/accuracy_6_of_7‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02850877307355404 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 _step‚894 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319833755493 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4953373372554779 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5443077087402344 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02219662256538868 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013609879650175571 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46748360991477966 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.43820664286613464 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1924210637807846 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.42193594574928284 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5073891282081604 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14646103978157043 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11929824948310852 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22221305966377258 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15102343261241913 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14323411881923676 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 6"Purity/0FRt0SRqqt*FBt/event_purity‚0.8153036236763 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12260536849498749 particle/accuracy_1_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 jet/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4976148307323456 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +particle/sensitivity‚0.3261105716228485 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17730247974395752 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 particle/accuracy_6_of_6‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.44034600257873535 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4096725881099701 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.563218355178833 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4673842489719391 *jet/accuracy_2_of_4‚0.3684903383255005 particle/accuracy_3_of_4‚1 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.289814829826355 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.027300916612148285 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7398809194564819 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5794482827186584 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2636807858943939 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026729561388492584 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44951316714286804 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2085375189781189 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5604166984558105 particle/accuracy_2_of_3‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6501566171646118 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 -jet/accuracy_5_of_5‚0.0026881720405071974 *particle/specificity‚0.951553225517273 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34077510237693787 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1855158656835556 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15977101027965546 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4528508484363556 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16329120099544525 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1450265794992447 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5526841282844543 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4488506019115448 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3998498022556305 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11884669214487076 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5228611826896667 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3961279094219208 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 +jet/accuracy_1_of_1‚0.47304609417915344 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016512228175997734 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16426487267017365 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6140351295471191 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20720787346363068 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2916666567325592 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 )loss/val_total_loss‚41.14814647095561 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 +jet/accuracy_2_of_6‚0.43589746952056885 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48961666226387024 ,jet/accuracy_4_of_4‚0.005641418509185314 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8569929003715515 particle/accuracy_2_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33541956543922424 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11781811714172363 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_3_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7575757503509521 +jet/accuracy_2_of_3‚0.24746295809745789 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_5_of_6‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42596447467803955 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06226581335067749 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1427823007106781 particle/accuracy_1_of_1‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2916666567325592 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21968764066696167 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20552532374858856 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.43984678387641907 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5151978731155396 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_6‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0242468249052763 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03440660238265991 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_3_of_6‚0.1538461595773697 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 jet/accuracy_2_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5102682113647461 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03977314010262489 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5284140110015869 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.45861175656318665 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4216611087322235 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45167791843414307 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_4_of_4‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4617496728897095 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019198792055249214 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4673842489719391 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20942345261573792 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919359683990479 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10210384428501129 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4923475682735443 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.607291579246521 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.025193799287080765 particle/accuracy_5_of_5‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979721903800964 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2736004590988159 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012850729748606682 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6974282264709473 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34347444772720337 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5273220539093018 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16591784358024597 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13607747852802277 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909806251525879 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2460939884185791 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1150793582201004 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1633979231119156 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45448458194732666 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12586207687854767 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17831240594387054 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15029282867908478 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.040028978139162064 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 )jet/accuracy_2_of_5‚0.433563768863678 particle/accuracy_1_of_3‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3915032148361206 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5728175044059753 +jet/accuracy_2_of_2‚0.11052113026380539 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9930272102355957 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1788882315158844 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2768493890762329 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09880952537059784 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868131875991821 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_6‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3866012394428253 particle/accuracy_1_of_7‚1 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17132151126861572 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4948727786540985 particle/accuracy_1_of_2‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27196839451789856 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841078519821167 ("Purity/2FRt4SRqqt0FBt/event_purity‚0þtó’â'‚(:‡ ø‹äԪώ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.409' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vˆ ¹{ã'‚(:ð ø‹äÔÖƒ  network.sent‚ 2.6667831e+07! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.sda3.out‚ 387.50390625 disk.sda1.in‚0' disk.nvme0c0n1.out‚54965.8330078125" proc.memory.rssMB‚ 2810.9296875 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0$ proc.memory.availableMB‚2957.625 proc.cpu.threads‚17$ disk./.usageGB‚247.9116973876953 disk.rbd1.out‚35.75341796875 disk.sda1.out‚0 disk.sda2.out‚0 network.recv‚ 3.13229e+06 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚54965.8330078125 disk.dm-1.in‚ 0.1953125 disk.rbd0.out‚0$ memory_percent‚81.94808959960938* disk./.usagePercent‚16.637562023457335 disk.dm-1.out‚ 387.50390625 disk.sda.out‚ 387.50390625# disk.nvme0c0n1.in‚ 1187.62109375" disk.dm-0.out‚54965.8330078125 disk.drbd2266.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.156553268432617 disk.rbd1.in‚0 disk.drbd2080.out‚0 cpu‚25.36416077552912%£ )2ä'- _wandbruntime‚8536  _runtime‚8536•îC}Èå' _step‚895 & lr-AdamW/pg1‚8.051335285418288e-06 & lr-AdamW/pg2‚8.051335285418288e-06 trainer/global_step‚10899 % _timestamp‚1.7884134333456163e+09 _runtime‚8536.333045746ÇÓ3øÍæ'Ç _runtime‚8536.333045746 _step‚895 & lr-AdamW/pg1‚8.051335285418288e-06 & lr-AdamW/pg2‚8.051335285418288e-06 trainer/global_step‚10899 % _timestamp‚1.7884134333456163e+09ÿQÔ9c2ç'- _wandbruntime‚8536  _runtime‚8536ß>ŠÂKè'âO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158908998 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206051440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206051440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚703078897614336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚627591111999243 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364960440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_startx¤&î#/allocation.large_pool.allocated‚ 158909000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9589852626944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52268968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚635147644255744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884134333463385e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9141834476544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚644290153097728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206031402 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚635147644255744 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9141834476544 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144994083 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92724670 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚644289478732288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9108826837771 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644289478732288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚693489236745728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158908998 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144993636 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9142490839040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9589684808704 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206031402 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8536.334030876 _step‚896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52269413 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364940400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9142490839040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚10899 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚644290153097728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚635147662258688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚635147662258688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚618482285161472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚618482268122112 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693489212805632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92724668 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9108172400015 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364960440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158909000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9589852626944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52268968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚635147644255744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884134333463385e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9141834476544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚644290153097728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206031402 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚635147644255744 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9141834476544 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144994083 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92724670 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚644289478732288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9108826837771 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644289478732288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚693489236745728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158908998 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144993636 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9142490839040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9589684808704 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206031402 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8536.334030876 _step‚896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52269413 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364940400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9142490839040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚10899 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚644290153097728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚635147662258688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚635147662258688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚618482285161472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚618482268122112 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693489212805632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92724668 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9108172400015 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8537.035661864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚644346651198976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145006207 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9143310280192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9142653904384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚635203997294592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884134340475059e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364992936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9590526541312 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206069920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52273825 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚627646161087651 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚618537172252672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158923014 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635203997294592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9142653904384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158923016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚703141390084608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9108988834979 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚644347325577728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206049856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145006663 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚10899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚635204015297536 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚644347325577728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92732836 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0»*/ ì'ñM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚635204015297536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92732838 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚618537189292032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364972870 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158923016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206069920 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206049856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚627646832564975 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145006207 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9143310280192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9142653904384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚635203997294592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884134340475059e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364992936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9590526541312 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206069920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52273825 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚627646161087651 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚618537172252672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_trainYõ1?È_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158923014 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635203997294592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9142653904384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158923016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚703141390084608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9108988834979 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚644347325577728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206049856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145006663 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚10899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚635204015297536 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚644347325577728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92732836 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚897 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9109643272943 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8537.035661864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚644346651198976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647148116480512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145626679 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637965314201088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366567252 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637965314201088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647148116480512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637965296198144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9183476644864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621227491689472 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚647148790845952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93133070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52493176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206977492 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706204541466624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9632612602368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9149662177179 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159609800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206957454 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9183476644864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366587292 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93133068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚630376482389535 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚621227474650112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884134707887478e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9182820282368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚435 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145626244 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647148790845952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159609798 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159609798 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637965296198144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706204362291200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚630377153866651 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9183476644864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621227491689472 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚647148790845952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93133070 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52493176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206977492 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706204541466624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9632612602368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9149662177179 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159609800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206957454 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9183476644864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366587292 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93133068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚630376482389535 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚621227474650112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884134707887478e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9182820282368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚435 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145626244 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647148790845952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159609798 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159609798 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637965296198144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706204362291200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚630377153866651 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647148116480512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.onNŠâü_train_batch_start/inactive_split.all.allocated‚ 145626679 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637965314201088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2„±*Ÿ×2ú'- _wandbruntime‚8574  _runtime‚8574#À•„ò&û'ìM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚706266883466240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159623816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619788 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9150478612351 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9184296086016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚901 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚647205963325952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52498140 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚706267064725504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚8574.468074039 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206975908 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚10949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145639378 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93141236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619788 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚621282378780672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚696633424366592 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159623814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚426 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9183639710208 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145638952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366599722 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159623816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366599722 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9184296086016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183639710208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621282395820032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9633483039744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430F°^˜÷&ü'ñM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚706266883466240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159623816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619788 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9150478612351 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9184296086016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚901 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚647205963325952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52498140 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚706267064725504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚8574.468074039 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206975908 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚10949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145639378 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93141236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619788 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚621282378780672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚696633424366592 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159623814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚426 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9183639710208 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145638952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366599722 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159623816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366599722 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9184296086016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183639710208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621282395820032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9633483039744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430…JëõÍ2ý'- _wandbruntime‚8574  _runtime‚8574Ìæó~_þ'Ù epoch‚9 0loss/FRt4/assignment_loss‚2.2525527827117755 2loss/SRqqt3/assignment_loss‚1.0982769044581802 1loss/FBt3/assignment_loss‚0.11491038579266954 .loss/FRt2/detection_loss‚2.210751125081184 0loss/SRqqt1/detection_loss‚2.570916494497886 0loss/SRqqt2/detection_loss‚1.824619606461214 /loss/FBt1/detection_loss‚2.2823210755983987 0loss/FRt3/assignment_loss‚1.2472628485072743 2loss/SRqqt4/assignment_loss‚1.3387185782363469 0loss/FBt1/assignment_loss‚0.1675390645913358 3loss/FBt4/assignment_loss‚0.0034775322960090955 0loss/FRt1/assignment_loss‚2.6716201938688755 /loss/FRt1/detection_loss‚2.9080630373209715 .loss/FRt3/detection_loss‚1.804216325215318 /loss/FRt4/detection_loss‚2.5062883405581764 0loss/FRt2/assignment_loss‚2.4944525293328543 1loss/SRqqt3/detection_loss‚0.8808061631826254 .loss/FBt2/detection_loss‚2.204196435213089 _step‚902 % _timestamp‚1.7884134714820042e+09 /loss/FBt3/detection_loss‚0.6205484986934798 2loss/SRqqt1/assignment_loss‚1.2835373630200944 2loss/SRqqt2/assignment_loss‚1.2985597034187422 1loss/FBt2/assignment_loss‚0.21803170994389803 .loss/FBt4/detection_loss‚2.530310937336513 trainer/global_step‚10949 _runtime‚8574.469334288 1loss/SRqqt4/detection_loss‚1.6410278325731105 %loss/total_loss‚38.17300546791002Ò$vèdÿ'Þ epoch‚9 0loss/FRt4/assignment_loss‚2.2525527827117755 2loss/SRqqt3/assignment_loss‚1.0982769044581802 1loss/FBt3/assignment_loss‚0.11491038579266954 .loss/FRt2/detection_loss‚2.210751125081184 0loss/SRqqt1/detection_loss‚2.570916494497886 0loss/SRqqt2/detection_loss‚1.824619606461214 /loss/FBt1/detection_loss‚2.2823210755983987 0loss/FRt3/assignment_loss‚1.2472628485072743 2loss/SRqqt4/assignment_loss‚1.3387185782363469 0loss/FBt1/assignment_loss‚0.1675390645913358 3loss/FBt4/assignment_loss‚0.0034775322960090955 0loss/FRt1/assignment_loss‚2.6716201938688755 /loss/FRt1/detection_loss‚2.9080630373209715 .loss/FRt3/detection_loss‚1.804216325215318 /loss/FRt4/detection_loss‚2.5062883405581764 0loss/FRt2/assignment_loss‚2.4944525293328543 1loss/SRqqt3/detection_loss‚0.8808061631826254 .loss/FBt2/detection_loss‚2.204196435213089 _step‚902 % _timestamp‚1.7884134714820042e+09 /loss/FBt3/detection_loss‚0.6205484986934798 2loss/SRqqt1/assignment_loss‚1.2835373630200944 2loss/SRqqt2/assignment_loss‚1.2985597034187422 1loss/FBt2/assignment_loss‚0.21803170994389803 .loss/FBt4/detection_loss‚2.530310937336513 trainer/global_step‚10949 _runtime‚8574.469334288 1loss/SRqqt4/detection_loss‚1.6410278325731105 %loss/total_loss‚38.17300546791002†a2’€(‚(:‡ ¥ŒäÔÍ– gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºŽ (‚(:ô ¥ŒäÔ½æ–  disk.sda1.in‚0$ memory_percent‚81.95075988769531( disk.nvme0c0n1.out‚55109.59521484375 disk.drbd1716.out‚0 disk.rbd1.out‚35.75341796875 disk.dm-1.out‚ 388.91015625 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0# disk.dm-0.out‚55109.59521484375 disk.dm-1.in‚ 0.1953125 disk.drbd2080.in‚0% proc.memory.availableMB‚ 2957.1875 cpu‚25.3624938878564 network.recv‚ 3.151511e+06! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17 network.sent‚ 2.6907308e+07& disk.nvme0n1.out‚55109.59521484375 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0* disk./.usagePercent‚16.645619604590014 disk.rbd1.in‚0 disk.sda.out‚ 388.91015625 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 388.91015625 disk.sda1.out‚0 disk.sda2.out‚0$ disk./.usageGB‚248.0317611694336 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2810.9296875# disk.nvme0c0n1.in‚ 1187.62109375*ä ‚(‚(:… ´ŒäÔíèŒ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#&xÍ‚ƒ(‚(:÷ ´ŒäÔ㈩  disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚55153.57568359375 network.recv‚ 3.157572e+06 disk.sda.out‚ 389.5859375* disk./.usagePercent‚16.648541162570766 disk.sda1.out‚0$ disk./.usageGB‚248.0752944946289 disk.drbd2266.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.drbd2265.out‚0 cpu‚25.361936983004448 disk.sda2.out‚0 disk.dm-1.in‚ 0.1953125 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 2.7015327e+07 disk.sda3.out‚ 389.5859375 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda3.in‚ 0.1953125 disk.rbd1.out‚35.75341796875 disk.dm-1.out‚ 389.5859375 disk.drbd2080.out‚0 disk.sda.in‚ 0.1953125$ memory_percent‚81.94925785064697) proc.memory.availableMB‚ 2957.43359375* proc.memory.percent‚17.156553268432617 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2810.9296875 disk.sda2.in‚0( disk.nvme0c0n1.out‚55153.57568359375& disk.nvme0n1.out‚55153.57568359375 proc.cpu.threads‚17 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375|!}%’„(‚(:‡ ÃŒäÔ¹™ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.796' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.7592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07<õ·}…(‚(:ò ÃŒäÔˆ°¶ $ memory_percent‚81.94925785064697 proc.cpu.threads‚17 network.recv‚ 3.161703e+06 disk.sda2.in‚0 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.dm-0.out‚55202.69921875 disk.sda.out‚ 390.25390625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.out‚35.75341796875 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375! disk.nvme0n1.in‚ 1187.62109375) proc.memory.availableMB‚ 2957.43359375" proc.memory.rssMB‚ 2810.9296875 disk.rbd1.in‚0# disk.nvme0n1.out‚55202.69921875 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 390.25390625 network.sent‚ 2.7021401e+07* disk./.usagePercent‚16.651584068122744% disk./.usageGB‚248.12063598632812% disk.nvme0c0n1.out‚55202.69921875 disk.dm-1.out‚ 390.25390625 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚17.156553268432617 cpu‚25.361406697976854 disk.drbd1716.in‚0 disk.rbd0.out‚0}d2†(- _wandbruntime‚8610  _runtime‚8610[^ òLJ(Á $ _timestamp‚1.788413507722263e+09 _runtime‚8610.709727481 _step‚903 & lr-AdamW/pg1‚6.766804119350744e-06 & lr-AdamW/pg2‚6.766804119350744e-06 trainer/global_step‚10999Z2¬]̈(Æ & lr-AdamW/pg1‚6.766804119350744e-06 & lr-AdamW/pg2‚6.766804119350744e-06 trainer/global_step‚10999 $ _timestamp‚1.788413507722263e+09 _runtime‚8610.709727481 _step‚903‡˜<é´2‰(- _wandbruntime‚8610  _runtime‚8610¥‹”+2Š(âO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52718897 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52718451 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚633163195734059 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚640782966143488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699654286572032 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚650007428594176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9223806088192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640782948140544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207883506 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚640782966143488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9224462450688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytesÎåS¶.large_pool.allocated‚699654310512128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚709329842727424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623972698217472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93541470 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368194104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650006754228736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9675556155392 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207903544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207883506 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160310600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207903544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633162524256943 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160310598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709330042874368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9223806088192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 trainer/global_step‚10999 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650007428594176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚650006754228736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9675732362240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623972681178112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368214144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9190497516587 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9189843078831 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541468 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928ãžÊêí'‹(çO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9223806088192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 trainer/global_step‚10999 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650007428594176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚650006754228736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9675732362240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623972681178112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368214144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9190497516587 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9189843078831 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541468 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52718897 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52718451 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚633163195734059 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚640782966143488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699654286572032 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚650007428594176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9223806088192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640782948140544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207883506 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚640782966143488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9224462450688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699654310512128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚709329842727424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623972698217472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93541470 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368194104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650006754228736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9675556155392 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207903544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207883506 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160310600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207903544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633162524256943 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160310598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709330042874368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430ˆÔ®"y2Œ(- _wandbruntime‚8611  _runtime‚8611Íã‹7ò&(ìM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚8611.421323452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160324614 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚709392545400320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚633218244822467 % _timestamp‚1.7884135084332356e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207922024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚699715937309696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9191313951759 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650063926695424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93549638 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚640839301179392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9225281891840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160324616 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160324614 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 368246640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚640839319182336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207922024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699715961249792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160324616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 368226574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚640839301179392 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650064601074176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9190659513795 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚10999 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650064601074176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640839319182336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚905 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146272472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376ã6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚905 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146272472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚8611.421323452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160324614 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚709392545400320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚633218244822467 % _timestamp‚1.7884135084332356e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207922024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚699715937309696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9191313951759 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650063926695424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93549638 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚640839301179392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9225281891840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24–¯ ðá401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160324616 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160324614 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 368246640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚640839319182336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207922024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699715961249792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160324616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 368226574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚640839301179392 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650064601074176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9190659513795 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚10999 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650064601074176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640839319182336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚643600600082944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712455531700224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652865391976960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚643600618085888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643600600082944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702736847395328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9264791894016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369840996 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52942251 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161011398 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161011400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9265448256512 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161011400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52942682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9265448256512 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652865391976960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635949237601467 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161011398ìšž@í'›(çO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚643600600082944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712455531700224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652865391976960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚643600618085888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643600600082944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702736847395328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9264791894016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369840996 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52942251 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161011398 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161011400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9265448256512 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161011400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52942682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9265448256512 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652865391976960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635949237601467 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161011398 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230678418239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635948566124351 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369820956 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚652866066342400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9231494853203 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146904643 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9719537229824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161025414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208848076 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52947050 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonito¥dgzFr.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9719365230592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146905088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702798474192896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636004286689875 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161025414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52946607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚643656971124736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873492 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161025416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652923238822400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208828012 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚626772791836672 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369873492 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚8648.723697891 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9266267697664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚643656953121792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369853426 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853426 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208828012 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚643656953121792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702798498132992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚11049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265611321856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚643656971124736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚712518035362816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚652923238822400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652922564443648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9232149291167 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208848076 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636004958167199 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884135457357016e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9231494853203 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146904643 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9719537229824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161025414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208848076 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52947050 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9719365230592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146905088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702798474192896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636004286689875 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161025414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52946607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚643656971124736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873492 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161025416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652923238822400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208828012 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚626772791836672 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369873492 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚8648.723697891 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9266267697664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚643656953121792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369853426 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853426 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208828012 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚643656953121792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702798498132992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚11049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265611321856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚643656971124736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚712518035362816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚652923238822400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652922564443648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9232149291167 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208848076 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636004958167199 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884135457357016e+09í#0Ú2Ÿ(- _wandbruntime‚8648  _runtime‚8648·d"ÏR (Ì 0loss/FBt2/assignment_loss‚0.0406074063454677 1loss/SRqqt4/detection_loss‚1.5571913704520366 .loss/FBt2/detection_loss‚2.292583273516761 2loss/SRqqt1/assignment_loss‚1.6925004923061744 .loss/FBt1/detection_loss‚2.997272710005442 /loss/FRt2/assignment_loss‚2.694305313958062 1loss/SRqqt1/detection_loss‚2.2013117993200146 %loss/total_loss‚40.27911500100714 # _timestamp‚1.78841354573775e+09 0loss/FRt3/assignment_loss‚1.1363573748132456 .loss/FBt4/detection_loss‚2.870715538660685 /loss/FRt4/assignment_loss‚2.405897757521382 2loss/SRqqt3/assignment_loss‚0.8013314254284423 /loss/FRt4/detection_loss‚2.2012171033355923 /loss/FRt1/assignment_loss‚2.664601001246222 0loss/SRqqt2/detection_loss‚1.703085128428801 epoch‚9 0loss/FBt1/assignment_loss‚1.2435659253290698 1loss/FBt4/assignment_loss‚0.17299119226144435 .loss/FRt1/detection_loss‚2.996127258087027 1loss/SRqqt3/detection_loss‚0.8864120459184051 /loss/FBt3/detection_loss‚0.5771452638611663 _runtime‚8648.725104486 0loss/SRqqt2/assignment_loss‚1.40452249516856 .loss/FBt3/assignment_loss‚0.08246950381408 /loss/FRt3/detection_loss‚1.9245178296190242 trainer/global_step‚11049 _step‚910 /loss/FRt2/detection_loss‚2.3684452004037153 1loss/SRqqt4/assignment_loss‚1.363940591206316¶–÷ßW¡(Ñ 0loss/FRt3/assignment_loss‚1.1363573748132456 .loss/FBt4/detection_loss‚2.870715538660685 /loss/FRt4/assignment_loss‚2.405897757521382 2loss/SRqqt3/assignment_loss‚0.8013314254284423 /loss/FRt4/detection_loss‚2.2012171033355923 /loss/FRt1/assignment_loss‚2.664601001246222 0loss/SRqqt2/detection_loss‚1.703085128428801 epoch‚9 0loss/FBt1/assignment_loss‚1.2435659253290698 1loss/FBt4/assignment_loss‚0.17299119226144435 .loss/FRt1/detection_loss‚2.996127258087027 1loss/SRqqt3/detection_loss‚0.8864120459184051 /loss/FBt3/detection_loss‚0.5771452638611663 _runtime‚8648.725104486 0loss/SRqqt2/assignment_loss‚1.40452249516856 .loss/FBt3/assignment_loss‚0.08246950381408 /loss/FRt3/detection_loss‚1.9245178296190242 trainer/global_step‚11049 _step‚910 /loss/FRt2/detection_loss‚2.3684452004037153 1loss/SRqqt4/assignment_loss‚1.363940591206316 0loss/FBt2/assignment_loss‚0.0406074063454677 1loss/SRqqt4/detection_loss‚1.5571913704520366 .loss/FBt2/detection_loss‚2.292583273516761 2loss/SRqqt1/assignment_loss‚1.6925004923061744 .loss/FBt1/detection_loss‚2.997272710005442 /loss/FRt2/assignment_loss‚2.694305313958062 1loss/SRqqt1/detection_loss‚2.2013117993200146 %loss/total_loss‚40.27911500100714 # _timestamp‚1.78841354573775e+09ŽïPæ’¢(‚(:‡ ðŒäÔí¥’ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]|£(‚(:ñ ðŒäÔ»¬€  disk.sda.out‚ 391.86328125" proc.memory.rssMB‚ 2810.9296875" disk.dm-0.out‚55326.3017578125 disk.sda3.in‚ 0.1953125 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd1.out‚35.75341796875% disk.nvme0n1.out‚55326.3017578125 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0# memory_percent‚81.9498062133789 disk.sda1.in‚0 disk.sda3.out‚ 391.86328125 disk.drbd2266.out‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 2957.34375* proc.memory.percent‚17.156553268432617 cpu‚25.35978883662805 proc.cpu.threads‚17 network.sent‚ 2.7141666e+07* disk./.usagePercent‚16.654302100113775 network.recv‚ 3.1769e+06 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 391.86328125 disk.drbd2080.in‚0% disk./.usageGB‚248.16113662719727 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125# disk.nvme0c0n1.in‚ 1187.62109375' disk.nvme0c0n1.out‚55326.3017578125 disk.drbd1716.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375»ÒȤ(‚(:’ ÿŒäÔ¾çó gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.529' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.105799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Çû„¥(‚(:ù ÿŒäÔù©ð  disk.rbd0.out‚0 network.sent‚ 2.7250435e+07 disk.drbd2265.out‚0 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 disk.rbd1.out‚35.75341796875 disk.dm-0.in‚ 1187.62109375 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 2956.90234375# disk.dm-0.out‚55373.99853515625 disk.sda3.out‚ 392.27734375 disk.drbd2266.out‚0 disk.drbd2080.out‚0 cpu‚25.359255203553456 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 392.27734375 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2810.9296875 network.recv‚ 3.183814e+06# disk./.usageGB‚248.174560546875 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.156553268432617 disk.drbd1716.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.out‚ 392.27734375 disk.sda2.out‚0* disk./.usagePercent‚16.655202990626766& disk.nvme0n1.out‚55373.99853515625 disk.sda3.in‚ 0.1953125( disk.nvme0c0n1.out‚55373.99853515625 disk.rbd1.in‚0$ memory_percent‚81.95250034332275'0Ç2¦(- _wandbruntime‚8684  _runtime‚8684¢ˆÒkʧ(Ä _step‚911 ' lr-AdamW/pg1‚5.5912702943492725e-06 ' lr-AdamW/pg2‚5.5912702943492725e-06 trainer/global_step‚11099 % _timestamp‚1.7884135819358497e+09 _runtime‚8684.923270134ðãáÄϨ(É ' lr-AdamW/pg1‚5.5912702943492725e-06 ' lr-AdamW/pg2‚5.5912702943492725e-06 trainer/global_step‚11099 % _timestamp‚1.7884135819358497e+09 _runtime‚8684.923270134 _step‚911Âœ|2©(- _wandbruntime‚8684  _runtime‚8684΢ Z ª(âO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209735610 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9761341500928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655724704090624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638735279468875 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚655724029725184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655724029725184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53166689 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9271513757647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9272168195403 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371447808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚646418252025344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.smal B *Žl_pool.allocated‚ 9306434062336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161712198 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94358268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚646418252025344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715580893597696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9761509319168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646418270028288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚715580701839360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚6464182700282883·x]í'«(çO BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53166689 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9271513757647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9272168195403 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371447808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚646418252025344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9306434062336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161712198 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94358268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚646418252025344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715580893597696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9761509319168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646418270028288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚715580701839360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚646418270028288 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209735610 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9761341500928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655724704090624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638735279468875 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚655724029725184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655724029725184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424È-Á ’¬(‚(:‡ ŽäÔÊŒ„ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÔI ~­(‚(:ó Žäԧܛ  disk.rbd1.out‚35.83935546875$ disk.nvme0n1.out‚55419.380859375 disk.sda.out‚ 392.94140625) proc.memory.availableMB‚ 2993.93359375* proc.memory.percent‚17.156553268432617$ disk./.usageGB‚248.1927604675293 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd2266.in‚0$ memory_percent‚81.72647953033447* disk./.usagePercent‚16.656424402572625 disk.drbd2080.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.out‚ 392.94140625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 392.94140625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2810.9296875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 cpu‚25.358726512876572 disk.rbd0.out‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚55419.380859375 disk.dm-0.in‚ 1187.62109375! disk.dm-0.out‚55419.380859375 network.sent‚ 2.725637e+07 network.recv‚ 3.188159e+06 disk.drbd2080.in‚0éV‡t2®(- _wandbruntime‚8685  _runtime‚8685ïOTIñ&¯(ëM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161726214 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307253503488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209754064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9272984630575 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788413582638884e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209774128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9306597127680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646474605064192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371480278 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9272330192611 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161726216 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚646474605064192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147537571 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53171133 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371500344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646474623067136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚638791000034607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209754064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚655781202191872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚11099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371480278 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161726214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161726216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚629518015404032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9306597127680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚715643403253760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚629517998364672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94366438 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚655781876570624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚638790328557283 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705881035016192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9762210918400 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚8685.626764039 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94366436 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _step‚913 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147537132 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0)>Â>!#°(ðM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161726214 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307253503488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209754064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9272984630575 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788413582638884e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209774128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9306597127680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646474605064192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371480278 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9272330192611 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161726216 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚646474605064192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147537571 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53171133 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371500344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646474623067136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚638791000034607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209754064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚655781202191872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚11099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371480278 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161726214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161726216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚629518015404032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9306597127680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚715643403253760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚629517998364672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94366438 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚655781876570624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚638790328557283 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705881035016192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9762210918400 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚8685.626764039 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94366436 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _step‚913 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147537132 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928‘q&µW2±(- _wandbruntime‚8685  _runtime‚86853 öZ²(Ô 0loss/FBt1/assignment_loss‚1.4443684977595694 _runtime‚8685.628052922 /loss/FRt1/assignment_loss‚3.159963774681091 0loss/FRt2/assignment_loss‚3.0000826631273543 /loss/FRt2/detection_loss‚2.3709937572479247 .loss/FBt1/detection_loss‚4.840513467788696 /loss/FBt2/detection_loss‚10.829018354415894 _step‚914 2loss/SRqqt2/assignment_loss‚1.4715258947950094 0loss/SRqqt4/assignment_loss‚1.27220128994921 1loss/FBt3/assignment_loss‚0.06150554760491761 1loss/SRqqt3/detection_loss‚0.8471856198766652 0loss/FBt3/detection_loss‚0.49047583831151004 /loss/FRt4/assignment_loss‚2.861400865778631 2loss/FBt2/assignment_loss‚0.008149584755301476 -loss/FRt4/detection_loss‚2.51642144760307 0loss/SRqqt1/detection_loss‚2.223537243737115 2loss/FBt4/assignment_loss‚0.031473645227379166 0loss/SRqqt2/detection_loss‚1.982033984141147 /loss/FBt4/detection_loss‚1.5766124352812767 2loss/SRqqt3/assignment_loss‚0.9196554122985724 /loss/FRt1/detection_loss‚2.6934326767921446 /loss/FRt3/detection_loss‚1.3498939909040928 1loss/SRqqt4/detection_loss‚1.8616519965555356 %loss/total_loss‚50.18330961878539 epoch‚9 trainer/global_step‚11099 # _timestamp‚1.78841358264068e+09 /loss/FRt3/assignment_loss‚0.715119409774031 2loss/SRqqt1/assignment_loss‚1.6560922203792465©Fø7_³(Ù /loss/FRt1/assignment_loss‚3.159963774681091 0loss/FRt2/assignment_loss‚3.0000826631273543 /loss/FRt2/detection_loss‚2.3709937572479247 .loss/FBt1/detection_loss‚4.840513467788696 /loss/FBt2/detection_loss‚10.829018354415894 _step‚914 2loss/SRqqt2/assignment_loss‚1.4715258947950094 0loss/SRqqt4/assignment_loss‚1.27220128994921 1loss/FBt3/assignment_loss‚0.06150554760491761 1loss/SRqqt3/detection_loss‚0.8471856198766652 0loss/FBt3/detection_loss‚0.49047583831151004 /loss/FRt4/assignment_loss‚2.861400865778631 2loss/FBt2/assignment_loss‚0.008149584755301476 -loss/FRt4/detection_loss‚2.51642144760307 0loss/SRqqt1/detection_loss‚2.223537243737115 2loss/FBt4/assignment_loss‚0.031473645227379166 0loss/SRqqt2/detection_loss‚1.982033984141147 /loss/FBt4/detection_loss‚1.5766124352812767 2loss/SRqqt3/assignment_loss‚0.9196554122985724 /loss/FRt1/detection_loss‚2.6934326767921446 /loss/FRt3/detection_loss‚1.3498939909040928 1loss/SRqqt4/detection_loss‚1.8616519965555356 %loss/total_loss‚50.18330961878539 epoch‚9 trainer/global_step‚11099 # _timestamp‚1.78841358264068e+09 /loss/FRt3/assignment_loss‚0.715119409774031 2loss/SRqqt1/assignment_loss‚1.6560922203792465 0loss/FBt1/assignment_loss‚1.4443684977595694 _runtime‚8685.628052922’ÀÐé ´(‚(:’ äÔþ¦û gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚131.907' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.381400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ö61€µ(‚(:õ äÔéë‹  disk.drbd2266.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚55468.94873046875 disk.dm-1.out‚ 393.4296875% disk./.usageGB‚248.20249938964844 disk.sda1.out‚0 disk.drbd2265.in‚0 network.recv‚ 3.194978e+06 disk.drbd2080.in‚0 disk.rbd1.in‚0$ memory_percent‚81.72845840454102' proc.memory.availableMB‚ 2993.609375* disk./.usagePercent‚16.657077989807544 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚35.90185546875& disk.nvme0n1.out‚55468.94873046875 disk.dm-0.in‚ 1187.62109375 proc.cpu.threads‚17 disk.sda3.in‚ 0.1953125 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375 cpu‚25.35816787001453 disk.sda3.out‚ 393.4296875 disk.drbd2266.in‚0 disk.sda2.in‚0# disk.dm-0.out‚55468.94873046875" proc.memory.rssMB‚ 2810.9296875 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.sda.out‚ 393.4296875* proc.memory.percent‚17.156553268432617 network.sent‚ 2.7354094e+07/õ ¶(‚(:… ¬äÔ¿º³ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l(­{·(‚(:ð ¬äÔÞå³  disk.drbd2266.in‚0' proc.memory.availableMB‚ 3012.328125" proc.memory.rssMB‚ 2810.9296875 disk.rbd0.out‚0 disk.sda1.out‚0) disk./.usagePercent‚16.65757720449931 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0$ memory_percent‚81.61420822143555 disk.dm-0.in‚ 1187.62109375 disk.drbd2080.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚35.90185546875 disk.sda.in‚ 0.1953125 network.sent‚ 2.7451444e+07 network.recv‚ 3.201356e+06 disk.drbd2265.out‚0* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17$ disk./.usageGB‚248.2099380493164# disk.dm-0.out‚55526.66552734375 disk.drbd2266.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚55526.66552734375 disk.sda.out‚ 394.171875& disk.nvme0n1.out‚55526.66552734375 disk.rbd0.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 cpu‚25.35760853227147 disk.dm-1.out‚ 394.171875 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 394.171875:¹+2¸(- _wandbruntime‚8722  _runtime‚8722c·È¹( & lr-AdamW/pg2‚5.044687165022909e-06 trainer/global_step‚11149 % _timestamp‚1.7884136193986447e+09 _runtime‚8722.386116239 _step‚915 & lr-AdamW/pg1‚5.044687165022909e-06MÆ\ͺ(Ç % _timestamp‚1.7884136193986447e+09 _runtime‚8722.386116239 _step‚915 & lr-AdamW/pg1‚5.044687165022909e-06 & lr-AdamW/pg2‚5.044687165022909e-06 trainer/global_step‚11149“N΀a2»(- _wandbruntime‚8722  _runtime‚8722­¡£è'¼(âO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53393408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚708901897221632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚641521321336283 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94766668 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210681700 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658582667473408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658582667473408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9313003534811 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9346763505664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658583341838848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649235903967744 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9346763505664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9347419868160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9347419868160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373094700 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148159621 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚641520649859167 trainer/global_step‚11149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373074660 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148160078 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚916 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162412998 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658582667473408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9313003534811 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9346763505664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658583341838848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649235903967744 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9346763505664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9347419868160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9347419868160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373094700 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148159621 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚641520649859167 trainer/global_step‚11149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373074660 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148160078 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚916 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162412998 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53393408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚708901897221632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚641521321336283 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94766668 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210681700 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658582667473408 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9347582933504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9313819969983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚632263221932032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9805480628736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148172717 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649292275009536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148172275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94774836 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚632263204892672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚658640514318848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9313165532019 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162427014 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚718768864977408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segmeGd òJnt.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658639839940096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚708963571899392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚641576370424691 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94774838 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127196 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210680116 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 373107130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162427016 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162427016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9348239309312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9347582933504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9805317018112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚641577041902015 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9347582933504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9313819969983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚632263221932032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9805480628736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148172717 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649292275009536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148172275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94774836 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚632263204892672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚658640514318848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9313165532019 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162427014 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚718768864977408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658639839940096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚708963571899392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚641576370424691 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94774838 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127196 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210680116 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 373107130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162427016 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162427016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9348239309312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9347582933504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9805317018112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414•éj2Á(- _wandbruntime‚8723  _runtime‚8723wxâ]Â(× _step‚918 /loss/FRt3/assignment_loss‚1.770802129805088 2loss/SRqqt4/assignment_loss‚1.2976899241718154 1loss/FBt2/assignment_loss‚0.34420559857971966 0loss/SRqqt4/detection_loss‚1.702843814001729 % _timestamp‚1.7884136200985813e+09 0loss/FRt1/assignment_loss‚3.2423054377237954 /loss/FRt2/assignment_loss‚2.567002665705797 2loss/SRqqt2/assignment_loss‚1.3238476924184295 .loss/FBt4/detection_loss‚2.401988203326861 /loss/FRt4/assignment_loss‚2.455845831296383 0loss/FBt1/assignment_loss‚0.5380814022306973 1loss/FBt3/assignment_loss‚0.09631378618538018 1loss/SRqqt3/detection_loss‚1.0887737910626298 /loss/FBt3/detection_loss‚0.5415144653128624 trainer/global_step‚11149 1loss/SRqqt1/detection_loss‚2.2249358998404607 .loss/FRt4/detection_loss‚2.544523924207076 &loss/total_loss‚40.867043654851386 epoch‚9 _runtime‚8723.085945638 2loss/SRqqt3/assignment_loss‚0.8228852015721929 /loss/FRt2/detection_loss‚2.1495154481439136 /loss/FRt3/detection_loss‚1.8375209329184146 /loss/FBt1/detection_loss‚2.4508471488952637 /loss/FBt2/detection_loss‚2.8448723554611206 .loss/FRt1/detection_loss‚2.922419086650566 2loss/SRqqt1/assignment_loss‚1.4990713866013619 1loss/FBt4/assignment_loss‚0.31578025359092976 1loss/SRqqt2/detection_loss‚1.8834572751488952œÐsãbÃ(Ü /loss/FBt1/detection_loss‚2.4508471488952637 /loss/FBt2/detection_loss‚2.8448723554611206 .loss/FRt1/detection_loss‚2.922419086650566 2loss/SRqqt1/assignment_loss‚1.4990713866013619 1loss/FBt4/assignment_loss‚0.31578025359092976 1loss/SRqqt2/detection_loss‚1.8834572751488952 _step‚918 /loss/FRt3/assignment_loss‚1.770802129805088 2loss/SRqqt4/assignment_loss‚1.2976899241718154 1loss/FBt2/assignment_loss‚0.34420559857971966 0loss/SRqqt4/detection_loss‚1.702843814001729 % _timestamp‚1.7884136200985813e+09 0loss/FRt1/assignment_loss‚3.2423054377237954 /loss/FRt2/assignment_loss‚2.567002665705797 2loss/SRqqt2/assignment_loss‚1.3238476924184295 .loss/FBt4/detection_loss‚2.401988203326861 /loss/FRt4/assignment_loss‚2.455845831296383 0loss/FBt1/assignment_loss‚0.5380814022306973 1loss/FBt3/assignment_loss‚0.09631378618538018 1loss/SRqqt3/detection_loss‚1.0887737910626298 /loss/FBt3/detection_loss‚0.5415144653128624 trainer/global_step‚11149 1loss/SRqqt1/detection_loss‚2.2249358998404607 .loss/FRt4/detection_loss‚2.544523924207076 &loss/total_loss‚40.867043654851386 epoch‚9 _runtime‚8723.085945638 2loss/SRqqt3/assignment_loss‚0.8228852015721929 /loss/FRt2/detection_loss‚2.1495154481439136 /loss/FRt3/detection_loss‚1.8375209329184146–C"*Ä(‚(:… »äÔþš£ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.94' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.188 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ\-{Å(‚(:ð »äÔñù½  disk.drbd2265.out‚0 network.sent‚ 2.7457535e+07$ disk./.usageGB‚248.1860466003418 disk.rbd0.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚55566.08349609375 disk.dm-1.out‚ 394.78125 disk.drbd2080.out‚0) proc.memory.availableMB‚ 3011.69921875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0* proc.memory.percent‚17.156553268432617 disk.sda1.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd1716.out‚0# disk.dm-0.out‚55566.08349609375$ memory_percent‚81.61804676055908 cpu‚25.35710955499835 network.recv‚ 3.205778e+06( disk.nvme0c0n1.out‚55566.08349609375 disk.rbd1.in‚0 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 394.78125 disk.rbd1.out‚35.93408203125! disk.nvme0n1.in‚ 1187.62109375 disk.sda3.out‚ 394.78125* disk./.usagePercent‚16.655973829312362 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 ²6—’Æ(‚(:‡ ÊäÔÄîœ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.734' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?´šu~Ç(‚(:ó ÊäÔëîŠ  disk.sda1.out‚0 disk.drbd2080.out‚0# disk.nvme0c0n1.in‚ 1187.62109375( disk.nvme0c0n1.out‚55612.96240234375 disk.drbd2265.in‚0$ memory_percent‚81.61506652832031* proc.memory.percent‚17.156553268432617 cpu‚25.35658666908993# disk.dm-0.out‚55612.96240234375 disk.drbd2266.out‚0% disk./.usageGB‚248.20160675048828 disk.sda3.out‚ 395.3828125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚35.93408203125 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0% proc.memory.availableMB‚ 3012.1875 disk.sda2.in‚0 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2810.9296875 network.sent‚ 2.7566089e+07 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 395.3828125 disk.rbd0.out‚0* disk./.usagePercent‚16.657018084044527 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚55612.96240234375 disk.sda1.in‚0 network.recv‚ 3.212611e+06 disk.sda.out‚ 395.3828125 disk.sda3.in‚ 0.1953125¤òÂÕ2È(- _wandbruntime‚8759  _runtime‚8759¹J…ÈÉ( _step‚919 & lr-AdamW/pg1‚4.525713677309312e-06 & lr-AdamW/pg2‚4.525713677309312e-06 trainer/global_step‚11199 % _timestamp‚1.7884136566065905e+09 _runtime‚8759.594065614‘¨ƒ)ÍÊ(Ç & lr-AdamW/pg1‚4.525713677309312e-06 & lr-AdamW/pg2‚4.525713677309312e-06 trainer/global_step‚11199 % _timestamp‚1.7884136566065905e+09 _runtime‚8759.594065614 _step‚919—Ð…[Ÿ2Ë(- _wandbruntime‚8759  _runtime‚8759¼©Zè'Ì(âO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9388405673984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374721552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53618625 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163113798 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374701512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚661441979587072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721832163951104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711984458044928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚644307363203691 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚661441979587072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9388405673984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148793228 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9353184436463 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661441305221632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53618160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9353838874219 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211587714 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211607752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9847705906176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652053555910144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211587714 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884136566073625e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9847531796480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711984434104832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148793695 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚11199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚652053573913088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652053555910144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661441305221632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚721831965901312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9387749311488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211607752|[Ö?iÍ(çO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661441305221632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53618160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9353838874219 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211587714 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211607752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9847705906176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652053555910144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211587714 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884136566073625e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9847531796480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711984434104832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148793695 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚11199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚652053573913088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652053555910144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661441305221632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚721831965901312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9387749311488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211607752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9388405673984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374721552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53618625 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163113798 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374701512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚661441979587072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721832163951104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711984458044928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚644307363203691 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚661441979587072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9388405673984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148793228 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9353184436463 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414˜¾ËÎ(‚(:’ ÙäÔ¨ô² gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚125.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.031200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#¨eX|Ï(‚(:ñ ÙäÔŠÎ’ % disk./.usageGB‚248.21790313720703! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.out‚ 396.203125 disk.sda.in‚ 0.1953125 disk.sda1.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚17.156553268432617 network.recv‚ 3.216422e+06& disk.nvme0n1.out‚55665.25537109375 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 cpu‚25.356034373868344 disk.dm-1.in‚ 0.1953125 disk.drbd2266.in‚0$ memory_percent‚81.61520957946777 network.sent‚ 2.7571516e+07( disk./.usagePercent‚16.6581117482308 disk.sda3.out‚ 396.203125 disk.drbd2080.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚55665.25537109375 disk.drbd2265.out‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 3012.1640625 disk.drbd2265.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.sda.out‚ 396.203125 disk.dm-0.in‚ 1187.62109375 disk.sda1.in‚0( disk.nvme0c0n1.out‚55665.25537109375" proc.memory.rssMB‚ 2810.9296875 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 35.9970703125·ÇÉ=2Ð(- _wandbruntime‚8760  _runtime‚8760N1u,ñ&Ñ(ëM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163127814 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211626232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788413657307014e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53623107 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95183236 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148805897 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚661499152067072 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148806345 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652109908948992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚644362412292099 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9388568739328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚635008411420672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163127816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚652109926951936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95183238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9389225115136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211626232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚712046108782592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211606168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652109908948992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9354000871427 _runtime‚8760.295115433 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚661498477688320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721894657691136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9848372715008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733982 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211606168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163127816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721894457557504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚712046084842496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 _step‚921 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944ûp»‘ö&Ò(ðM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚661498477688320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721894657691136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9848372715008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733982 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211606168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163127816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721894457557504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚712046084842496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 _step‚921 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163127814 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211626232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788413657307014e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53623107 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95183236 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148805897 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚661499152067072 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148806345 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652109908948992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚644362412292099 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9388568739328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚635008411420672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163127816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚652109926951936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95183238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9389225115136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211626232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚712046108782592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211606168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652109908948992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9354000871427 _runtime‚8760.295115433™ðPw2Ó(- _wandbruntime‚8760  _runtime‚8760ì.”Ê]Ô(× 1loss/SRqqt2/detection_loss‚1.7942007056073002 1loss/SRqqt4/detection_loss‚1.7166658287698573 1loss/FBt3/assignment_loss‚0.13412237882162073 0loss/SRqqt3/detection_loss‚0.925889141112566 1loss/SRqqt2/assignment_loss‚1.739934664118387 /loss/FBt4/detection_loss‚3.2825457572937013 2loss/SRqqt1/assignment_loss‚1.5188437398937014 trainer/global_step‚11199 .loss/FBt1/detection_loss‚2.458436449368795 .loss/FBt2/detection_loss‚2.747658928235372 %loss/total_loss‚39.79801413890422 1loss/SRqqt3/assignment_loss‚0.651411242724862 2loss/FBt4/assignment_loss‚0.001315915072336793 /loss/FRt3/detection_loss‚1.3623485618777442 0loss/SRqqt1/detection_loss‚1.928679194715288 0loss/FRt2/assignment_loss‚2.8185432977146574 0loss/FRt3/assignment_loss‚0.9925921758015951 0loss/FRt4/assignment_loss‚2.8752456578341397 2loss/SRqqt4/assignment_loss‚1.3570346873253585 0loss/FBt2/assignment_loss‚0.3890411745524034 0loss/FBt3/detection_loss‚0.48083387041212217 /loss/FRt1/assignment_loss‚2.565796922892332 /loss/FRt4/detection_loss‚2.7563157889879113 _runtime‚8760.296730101 _step‚922 0loss/FBt1/assignment_loss‚0.1435976349748671 epoch‚9 % _timestamp‚1.7884136573093123e+09 /loss/FRt1/detection_loss‚2.6803833190351725 /loss/FRt2/detection_loss‚2.4765771017621323ág|Õ(Ü .loss/FBt1/detection_loss‚2.458436449368795 .loss/FBt2/detection_loss‚2.747658928235372 %loss/total_loss‚39.79801413890422 1loss/SRqqt3/assignment_loss‚0.651411242724862 2loss/FBt4/assignment_loss‚0.001315915072336793 /loss/FRt3/detection_loss‚1.3623485618777442 0loss/SRqqt1/detection_loss‚1.928679194715288 0loss/FRt2/assignment_loss‚2.818õ·ÁÃæ5432977146574 0loss/FRt3/assignment_loss‚0.9925921758015951 0loss/FRt4/assignment_loss‚2.8752456578341397 2loss/SRqqt4/assignment_loss‚1.3570346873253585 0loss/FBt2/assignment_loss‚0.3890411745524034 0loss/FBt3/detection_loss‚0.48083387041212217 /loss/FRt1/assignment_loss‚2.565796922892332 /loss/FRt4/detection_loss‚2.7563157889879113 _runtime‚8760.296730101 _step‚922 0loss/FBt1/assignment_loss‚0.1435976349748671 epoch‚9 % _timestamp‚1.7884136573093123e+09 /loss/FRt1/detection_loss‚2.6803833190351725 /loss/FRt2/detection_loss‚2.4765771017621323 1loss/SRqqt2/detection_loss‚1.7942007056073002 1loss/SRqqt4/detection_loss‚1.7166658287698573 1loss/FBt3/assignment_loss‚0.13412237882162073 0loss/SRqqt3/detection_loss‚0.925889141112566 1loss/SRqqt2/assignment_loss‚1.739934664118387 /loss/FBt4/detection_loss‚3.2825457572937013 2loss/SRqqt1/assignment_loss‚1.5188437398937014 trainer/global_step‚11199š"¯óÖ(‚(:’ èäÔèäþ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.269' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.653800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þjdä…×(‚(:ú èäԮ׌  proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 396.82421875 disk.sda.out‚ 396.82421875( disk.nvme0c0n1.out‚55728.72021484375 disk.drbd2266.in‚0$ memory_percent‚81.61542415618896* proc.memory.percent‚17.156553268432617# disk.nvme0c0n1.in‚ 1187.62109375! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚55728.72021484375 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 396.82421875 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚55728.72021484375 disk.drbd2265.in‚0 network.recv‚ 3.222916e+06 disk.rbd0.out‚0 disk.rbd1.out‚ 35.9970703125 disk.drbd2265.out‚0) proc.memory.availableMB‚ 3012.12890625 network.sent‚ 2.7668271e+07* disk./.usagePercent‚16.660859989110875% disk./.usageGB‚248.25885391235352 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚ 0.1953125 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2810.9296875 cpu‚25.355512990264682b©öL’Ø(‚(:‡ ÷äÔÊã¡ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.846' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þpÄ!ƒÙ(‚(:ø ÷äÔ”µ  disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚36.06005859375 disk.dm-1.out‚ 397.3515625 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2810.9296875 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 397.3515625 disk.drbd2266.in‚0 disk.rbd0.in‚0* disk./.usagePercent‚16.662482308855353 disk.sda1.in‚0 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚55797.48291015625# disk.nvme0c0n1.in‚ 1187.62109375) proc.memory.availableMB‚ 3011.87890625* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17( disk.nvme0c0n1.out‚55797.48291015625 cpu‚25.354962901407582 disk.sda2.in‚0& disk.nvme0n1.out‚55797.48291015625 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0$ memory_percent‚81.61695003509521 disk.drbd2265.out‚0 disk.sda.out‚ 397.3515625 network.sent‚ 2.7766458e+07 disk.sda2.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 network.recv‚ 3.229534e+06% disk./.usageGB‚248.28302764892578 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0ÏY"ÿ2Ú(- _wandbruntime‚8796  _runtime‚8796ØÆ×ÈÛ( _runtime‚8796.063661843 _step‚923 & lr-AdamW/pg1‚4.034457984505443e-06 & lr-AdamW/pg2‚4.034457984505443e-06 trainer/global_step‚11249 % _timestamp‚1.7884136930762331e+09¯J•|ÍÜ(Ç trainer/global_step‚11249 % _timestamp‚1.7884136930762331e+09 _runtime‚8796.063661843 _step‚923 & lr-AdamW/pg1‚4.034457984505443e-06 & lr-AdamW/pg2‚4.034457984505443e-06›4˜Hå2Ý(- _wandbruntime‚8796  _runtime‚8796Üaè'Þ(âO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚637698730857472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376348404 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9429391479808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9428735117312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376328364 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163814598 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664299942969856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163814598 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664300617335296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664299942969856 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149427652 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9890785897472 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163814600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚654871207852544 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149428102 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚11249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212533804 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9394674213627 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚715066994928128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚637698713818112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647092733593983Fn¦oí'ß(çO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚654871225855488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9428735117312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95583468 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9428735117312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376328364 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163814598 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664299942969856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163814598 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664300617335296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664299942969856 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149427652 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9890785897472 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163814600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚654871207852544 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149428102 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚11249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212533804 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9394674213627 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚715066994928128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚637698713818112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647092733593983 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚637698730857472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376348404 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9429391479808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884136937648911e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚654927578894336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163828616 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9430210920960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95591636 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9429554545152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9395490648799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380900 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚654927560891392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚654927578894336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664357789815296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚654927560891392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891470482944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚925 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376360834 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376360834 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚715128645665792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212552284 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394836210835 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95591638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212532220 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725020092208640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚11249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149440268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725020290245120 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚647148454159507 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149440727 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9891644579328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚8796.752975716 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637753617948672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚637753634988032 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163828614 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380900 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚715128621725696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E÷&â(ñM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95591638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212532220 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725020092208640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚11249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149440268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725020290245120 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚647148454159507 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149440727 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9891644579328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚8796.752975716 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637753617948672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚637753634988032 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163828614 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380900 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚715128621725696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884136937648911e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚654927578894336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163828616 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9430210920960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95591636 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9429554545152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9395490648799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380900 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚654927560891392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚654927578894336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664357789815296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚654927560891392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891470482944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚925 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376360834 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376360834 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚715128645665792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212552284 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394836210835 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872y,£2ã(- _wandbruntime‚8796  _runtime‚8796šÝÇ1[ä(Õ 0loss/SRqqt1/assignment_loss‚2.37793997079134 /loss/FRt3/detection_loss‚1.9646343064804872 /loss/FBt1/detection_loss‚2.7056151926517487 /loss/FRt4/detection_loss‚1.9370371249684117 _runtime‚8796.753896254 /loss/FRt1/assignment_loss‚2.877773393284191 1loss/FBt4/assignment_loss‚0.36005294218921335 /loss/FRt1/detection_loss‚3.1840053878047248 2loss/SRqqt4/assignment_loss‚1.1528119620834958 1loss/SRqqt2/detection_loss‚1.5916393785452356 % _timestamp‚1.7884136937665303e+09 2loss/SRqqt3/assignment_loss‚0.9832938351727863 /loss/FBt2/detection_loss‚2.0388109298313366 %loss/total_loss‚40.27480837814736 trainer/global_step‚11249 /loss/FRt2/assignment_loss‚2.590498289336329 .loss/FRt2/detection_loss‚2.105991000077433 1loss/SRqqt1/detection_loss‚3.2826636910438536 1loss/SRqqt3/detection_loss‚1.1766143837111427 0loss/SRqqt4/detection_loss‚1.383807719781481 0loss/FBt3/detection_loss‚0.46752697083478173 /loss/FBt4/detection_loss‚2.2124468088150024 /loss/FRt3/assignment_loss‚1.417493889066908 2loss/SRqqt2/assignment_loss‚1.2191770268338067 0loss/FBt1/assignment_loss‚0.3588154681128799 0loss/FBt2/assignment_loss‚0.5062455154873211 epoch‚9 _step‚926 /loss/FRt4/assignment_loss‚2.211001230959307 1loss/FBt3/assignment_loss‚0.16891196028414457å2`å(Ú 2loss/SRqqt4/assignment_loss‚1.1528119620834958 1loss/SRqqt2/detection_loss‚1.5916393785452356 % _timestamp‚1.7884136937665303e+09 2loss/SRqqt3/assignment_loss‚0.9832938351727863 /loss/FBt2/detection_loss‚2.0388109298313366 %loss/total_loss‚40.27480837814736 trainer/global_step‚11249 /loss/FRt2/assignment_loss‚2.590498289336329 .loss/FRt2/detection_loss‚2.105991000077433 1loss/SRqqt1/detection_loss‚3.2826636910438536 1loss/SRqqt3/detection_loss‚1.1766143837111427 0loss/SRqqt4/detection_loss‚1.383807719781481 0loss/FBt3/detection_loss‚0.46752697083478173 /loss/FBt4/detection_loss‚2.2124468088150024 /loss/FRt3/assignment_loss‚1.417493889066908 2loss/SRqqt2/assignment_loss‚1.2191770268338067 0loss/FBt1/assignment_loss‚0.3588154681128799 0loss/FBt2/assignment_loss‚0.5062455154873211 epoch‚9 _step‚926 /loss/FRt4/assignment_loss‚2.211001230959307 1loss/FBt3/assignment_loss‚0.16891196028414457 0loss/SRqqt1/assignment_loss‚2.37793997079134 /loss/FRt3/detection_loss‚1.9646343064804872 /loss/FBt1/detection_loss‚2.7056151926517487 /loss/FRt4/detection_loss‚1.9370371249684117 _runtime‚8796.753896254 /loss/FRt1/assignment_loss‚2.877773393284191 1loss/FBt4/assignment_loss‚0.36005294218921335 /loss/FRt1/detection_loss‚3.1840053878047248žÅ Ê~æ(‚(:’ †ŽäÔòƬ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.737' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.547399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”•»ýƒç(‚(:ø †ŽäÔäÄ­ & disk.nvme0n1.out‚55852.38916015625 disk.drbd2265.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2810.9296875 cpu‚25.354445145664183 disk.sda3.out‚ 398.0234375 disk.drbd2266.in‚0 disk.drbd2080.in‚0 network.sent‚ 2.7772383e+07 network.recv‚ 3.233943e+06% disk./.usageGB‚248.26776123046875 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 398.0234375 disk.drbd2265.out‚0 disk.rbd0.in‚0$ memory_percent‚81.61561489105225) proc.memory.availableMB‚ 3012.09765625 disk.rbd1.out‚36.14599609375! disk.nvme0n1.in‚ 1187.62109375 disk.sda2.out‚0( disk.nvme0c0n1.out‚55852.38916015625 disk.rbd1.in‚0# disk.dm-0.out‚55852.38916015625 disk.drbd2080.out‚0 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.sda.out‚ 398.0234375 disk.rbd0.out‚0 disk.drbd2266.out‚0* disk./.usagePercent‚16.661457766703325 disk.drbd1716.out‚0 disk.sda3.in‚ 0.1953125 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.156553268432617‰‰x’è(‚(:‡ •ŽäÔê½á gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚/ã†é(‚(:û •ŽäÔå÷ƒ * proc.memory.percent‚17.156553268432617" proc.memory.rssMB‚ 2810.9296875 disk.drbd1716.in‚0 disk.sda.out‚ 398.68359375 disk.drbd2266.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚55941.59130859375) proc.memory.availableMB‚ 3031.74609375* disk./.usagePercent‚16.662891920910138 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚36.20849609375 disk.drbd2080.out‚0 disk.dm-1.out‚ 398.68359375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1187.62109375% disk./.usageGB‚248.28913116455078! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚55941.59130859375# disk.dm-0.out‚55941.59130859375 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 network.sent‚ 2.7881315e+07 network.recv‚ 3.241169e+06 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 398.68359375 disk.sda1.in‚0$ memory_percent‚81.49569034576416 cpu‚25.353924907579373 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0 disk.sda2.out‚0e l:2ê(- _wandbruntime‚8833  _runtime‚8833JÇ’Êë(Ä ' lr-AdamW/pg1‚3.5710224635613796e-06 ' lr-AdamW/pg2‚3.5710224635613796e-06 trainer/global_step‚11299 % _timestamp‚1.7884137302109928e+09 _runtime‚8833.198437851 _step‚927•(Ö»Ïì(É ' lr-AdamW/pg1‚3.5710224635613796e-06 ' lr-AdamW/pg2‚3.5710224635613796e-06 trainer/global_step‚11299 % _timestamp‚1.7884137302109928e+09 _runtime‚8833.198437851 _step‚927Ÿža 2í(- _wandbruntime‚8833  _runtime‚8833A™cæ'î(àO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204341248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164515398 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164515400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54069689 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9470377285632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213439818 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 # _timestamp‚1.78841373021173e+09 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150062032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚728083115004928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚649879446938507 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150061557 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚640443920346112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚8833.199365543 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚728083319346176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚718149531811328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377955216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚640443937385472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚667158580718080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9435509553035 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164515400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚718149507871232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚649878775461391 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 377975256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164515398 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚667159255083520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667158580718080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213439818 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ñ&%ª.ï(åO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204341248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164515398 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164515400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54069689 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9470377285632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213439818 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 # _timestamp‚1.78841373021173e+09 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150062032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚728083115004928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚649879446938507 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150061557 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚640443920346112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚8833.199365543 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚728083319346176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚718149531811328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377955216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚640443937385472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚667158580718080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9435509553035 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164515400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚718149507871232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚649878775461391 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 377975256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164515398 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚667159255083520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667158580718080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213439818 aIõÆ2ð(- _wandbruntime‚8833  _runtime‚8833i!6ñ&ñ(ëM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚649934496026915 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164529414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96000036 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚667216427563520 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728145811149824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚649935167504239 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚11299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478336 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213458272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54074535 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884137309034307e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657745230836736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657745212833792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150074109 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚640498841516032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164529414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213478336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9436325988207 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 377987686 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9934452407296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657745212833792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚718211158608896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9470540350976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54074073 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9934628600832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚640498824476672 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150074109 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚640498841516032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164529414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213478336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9436325988207 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 377987686 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9934452407296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657745212833792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚718211158608896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9470540350976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54074073 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9934628600832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚640498824476672 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728145811149824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚649935167504239 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚11299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478336 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213458272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54074535 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884137309034307e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657745230836736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¡>lŒ2ó(- _wandbruntime‚8833  _runtime‚8833)6Œ¼^ô(Ø 2loss/SRqqt4/assignment_loss‚1.2174991217955036 .loss/FBt2/detection_loss‚2.224013012647629 0loss/FRt4/assignment_loss‚2.0400574008623757 2loss/SRqqt3/assignment_loss‚0.7996144517965149 .loss/FBt1/detection_loss‚2.182989488948475 /loss/FBt3/detection_loss‚0.6288123207191852 epoch‚9 0loss/SRqqt1/detection_loss‚2.651630454022309 _step‚930 2loss/SRqqt1/assignment_loss‚1.7778354795842335 2loss/SRqqt2/assignment_loss‚1.2382367244563424 1loss/FBt2/assignment_loss‚0.18904196777148172 1loss/FBt3/assignment_loss‚0.18905107656647444 .loss/FRt3/detection_loss‚1.780694324126903 0loss/SRqqt2/detection_loss‚1.390775715617033 /loss/FBt4/detection_loss‚2.6570793886979422 0loss/FRt2/assignment_loss‚1.9237154581967522 0loss/FRt3/assignment_loss‚1.3483702553079484 2loss/FBt4/assignment_loss‚0.023239325290584627 /loss/FRt4/detection_loss‚2.2395788782172734 1loss/SRqqt3/detection_loss‚0.8808626171667129 %loss/total_loss‚37.58138276750906 trainer/global_step‚11299 _runtime‚8833.892433226 1loss/FBt1/assignment_loss‚0.31989716540407576 .loss/FRt2/detection_loss‚2.042005504463233 % _timestamp‚1.7884137309051163e+09 /loss/FRt1/detection_loss‚3.0202754118612836 0loss/FRt1/assignment_loss‚3.1044037512370517 0loss/SRqqt4/detection_loss‚1.711703472751744CQÐcõ(Ý 0loss/FRt4/assignment_loss‚2.0400574008623757 2loss/SRqqt3/assignment_loss‚0.7996144517965149 .loss/FBt1/detection_loss‚2.182989488948475 /loss/FBt3/detection_loss‚0.6288123207191852 epoch‚9 0loss/SRqqt1/detection_loss‚2.651630454022309 _step‚930 2loss/SRqqt1/assignment_loss‚1.7778354795842335 2loss/SRqqt2/assignment_loss‚1.2382367244563424 1loss/FBt2/assignment_loss‚0.18904196777148172 1loss/FBt3/assignment_loss‚0.18905107656647444 .loss/FRt3/detection_loss‚1.780694324126903 0loss/SRqqt2/detection_loss‚1.390775715617033 /loss/FBt4/detection_loss‚2.6570793886979422 0loss/FRt2/assignment_loss‚1.9237154581967522 0loss/FRt3/assignment_loss‚1.3483702553079484 2loss/FBt4/assignment_loss‚0.023239325290584627 /loss/FRt4/detection_loss‚2.2395788782172734 1loss/SRqqt3/detection_loss‚0.8808626171667129 %loss/total_loss‚37.58138276750906 trainer/global_step‚11299 _runtime‚8833.892433226 1loss/FBt1/assignment_loss‚0.31989716540407576 .loss/FRt2/detection_loss‚2.042005504463233 % _timestamp‚1.7884137309051163e+09 /loss/FRt1/detection_loss‚3.0202754118612836 0loss/FRt1/assignment_loss‚3.1044037512370517 0loss/SRqqt4/detection_loss‚1.711703472751744 2loss/SRqqt4/assignment_loss‚1.2174991217955036 .loss/FBt2/detection_loss‚2.224013012647629¢.{­’ö(‚(:‡ ¤ŽäÔŽû‹ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.332' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.8664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]=Ô¶ƒ÷(‚(:ø ¤ŽäÔ˜¦Ž  disk.rbd1.out‚36.20849609375& disk.nvme0n1.out‚55985.88916015625 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 399.48046875 disk.drbd2080.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚55985.88916015625 disk.drbd1716.out‚0$ memory_percent‚81.49576187133789 disk.dm-0.in‚ 1187.62109375% disk./.usageGB‚248.30377960205078 disk.sda3.in‚ 0.1953125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 cpu‚25.353409882577218 proc.cpu.threads‚17 network.sent‚ 2.7886773e+07) disk./.usagePercent‚16.66387498984162 disk.sda2.out‚0' proc.memory.availableMB‚ 3031.734375" proc.memory.rssMB‚ 2810.9296875 disk.sda3.out‚ 399.48046875 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.sda.out‚ 399.48046875 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375* proc.memory.percent‚17.156553268432617# disk.dm-0.out‚55985.88916015625 disk.drbd2265.in‚0 network.recv‚ 3.245185e+06%’#ä’ø(‚(:‡ ³ŽäÔ›–„ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†k6‚ù(‚(:÷ ³ŽäÔˆõ‹ ( disk.nvme0c0n1.out‚56045.56787109375 disk.sda3.in‚ 0.1953125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0$ memory_percent‚81.49585723876953 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚36.24072265625& disk.nvme0n1.out‚56045.56787109375 network.sent‚ 2.7995482e+07 network.recv‚ 3.251721e+06 disk.drbd2080.in‚0# disk.dm-0.out‚56045.56787109375 disk.dm-1.out‚ 400.01171875 disk.drbd2265.out‚0* proc.memory.percent‚17.156553268432617) disk./.usagePercent‚16.66515118739981 cpu‚25.352897181092704! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125" proc.memory.rssMB‚ 2810.9296875 proc.cpu.threads‚17 disk.sda1.out‚0% disk./.usageGB‚248.32279586791992 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.out‚ 400.01171875& proc.memory.availableMB‚ 3031.71875 disk.rbd0.out‚0 disk.sda3.out‚ 400.01171875 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375Í´ºú(‚(:„ ÂŽäÔ¿­ð gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.p0ÃMÞÕowerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÍmÜû(‚(:ô ÂŽäÔÓñ©  disk.dm-1.out‚ 400.546875 network.sent‚ 2.8001392e+07 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375( disk.nvme0c0n1.out‚56081.81396484375 disk.drbd2265.out‚0* proc.memory.percent‚17.156553268432617 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚ 3.256144e+06% disk./.usageGB‚248.33517837524414 disk.sda3.out‚ 400.546875 disk.drbd2266.out‚0 disk.sda.out‚ 400.546875 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚36.24072265625 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0) proc.memory.availableMB‚ 3031.71484375" proc.memory.rssMB‚ 2810.9296875* disk./.usagePercent‚16.665982187855953 disk.drbd2266.in‚0# disk.dm-0.out‚56081.81396484375 disk.sda.in‚ 0.1953125 cpu‚25.35236249441386 disk.rbd0.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚56081.81396484375$ memory_percent‚81.49588108062744?OÔw2ü(- _wandbruntime‚8870  _runtime‚8870V™zlÊý(Ä ' lr-AdamW/pg1‚3.1355036937450917e-06 ' lr-AdamW/pg2‚3.1355036937450917e-06 trainer/global_step‚11349 % _timestamp‚1.7884137670824835e+09 _runtime‚8870.069971258 _step‚931ð›Ñ{Ïþ(É ' lr-AdamW/pg1‚3.1355036937450917e-06 ' lr-AdamW/pg2‚3.1355036937450917e-06 trainer/global_step‚11349 % _timestamp‚1.7884137670824835e+09 _runtime‚8870.069971258 _step‚931£K8M=2ÿ(- _wandbruntime‚8870  _runtime‚8870QHR@è'€)âO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 214365870 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660506529740288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652664817328799 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214365870 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660506511737344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670017218466304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9510706728960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 % _timestamp‚1.7884137670832963e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96400270 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721232044754432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9475690454687 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670017218466304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 379602108 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚643189143913472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96400268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670017892831744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165216200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚660506529740288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731208758044160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150697442 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643189126874112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54297172 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165216200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652665488805915 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9511363091456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚11349 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚660506511737344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165216198 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721232068694528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54296741 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚731208580965888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670017892831744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2‰œ–ní')çO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731208758044160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150697442 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643189126874112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54297172 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165216200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652665488805915 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9511363091456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚11349 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚660506511737344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165216198 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721232068694528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54296741 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚731208580965888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670017892831744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 214365870 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660506529740288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652664817328799 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214365870 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660506511737344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670017218466304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9510706728960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 % _timestamp‚1.7884137670832963e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96400270 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721232044754432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9475690454687 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670017218466304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 379602108 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚643189143913472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96400268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670017892831744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165216200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚660506529740288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440¤ä9m2‚)- _wandbruntime‚8870  _runtime‚8870Â?§ ò&ƒ)ìM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165230216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660562882779136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚933 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165230214 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9511526156800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634604 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96408436 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214404388 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _runtime‚8870.773312811 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660562864776192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96408438 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165230214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150709638 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643244031004672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721293719432192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721293695492096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11349 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150710076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9476506889651 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652721209371647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9977399467520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214384324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884137677851863e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 379614538 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9511526156800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54301202 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚731271094959616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9512182532608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670075065311744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379634604 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214404388 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670075065311744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670074390932992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0÷„i„)ñM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165230216 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150709638 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643244031004672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721293719432192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721293695492096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11349 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150710076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9476506889651 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652721209371647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9977399467520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214384324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884137677851863e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 379614538 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9511526156800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54301202 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚731271094959616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9512182532608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670075065311744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379634604 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214404388 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670075065311744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670074390932992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165230216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660562882779136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚933 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165230214 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9511526156800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634604 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96408436 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214404388 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _runtime‚8870.773312811 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660562864776192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96408438 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165230214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚663324163679744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96808670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215291922 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚663324181682688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672875856214528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552348897280 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165916998 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551692534784 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚672876530579968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96808668 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734333984177152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9516525794095 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 _runtime‚8906.621552643 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165917000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151333816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215291922 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9552348897280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9551692534784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚655451530673323 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚655450859196207 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645934350441472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚672875856214528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663324163679744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚724314605577728 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10019402539520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚645934333402112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚11399 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9551692534784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚655451530673323 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚655450859196207 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645934350441472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚672875856214528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663324163679744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚724314605577728 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10019402539520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚645934333402112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚11399 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚663324163679744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_mQÙèpool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96808670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215291922 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚663324181682688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672875856214528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552348897280 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165916998 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551692534784 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚672876530579968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96808668 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734333984177152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9516525794095 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 _runtime‚8906.621552643 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165917000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151333816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215291922¨ÁȾÔ2’)- _wandbruntime‚8907  _runtime‚8907aM Tò&“)ìM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚655507251239055 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645989237532672 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215330440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚663380534721536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10020422567936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9552511962624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9517996667023 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚724376232375296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553168338432 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165931014 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10020244277248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663380516718592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552511962624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165931016 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178290688 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645989254572032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 # _timestamp‚1.78841380431319e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚672933028681216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54529671 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165931016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151346507 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96816836 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚672933703059968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151346958 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734396476652544 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚663380534721536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381241390 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215310376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655506579761731 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165931014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚724376256315392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734396678883328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381241390 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8907.301147608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚663380516718592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚672933703059968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚937 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215310376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54530120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚11399 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FÉ­÷&”)ñM OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165931014 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10020244277248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663380516718592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552511962624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165931016 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178290688 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645989254572032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 # _timestamp‚1.78841380431319e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚672933028681216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54529671 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165931016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151346507 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96816836 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚672933703059968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151346958 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734396476652544 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚663380534721536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381241390 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215310376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655506579761731 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165931014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚724376256315392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734396678883328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381241390 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8907.301147608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚663380516718592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚672933703059968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚937 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215310376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54530120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚11399 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9552511962624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9517996667023 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚724376232375296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553168338432©: ÔÎ2•)- _wandbruntime‚8907  _runtime‚8907íDe^–)Ø % _timestamp‚1.7884138043148682e+09 0loss/FRt1/assignment_loss‚3.1201183259487153 2loss/SRqqt3/assignment_loss‚0.7803797069797987 1loss/FBt4/assignment_loss‚0.23987608101742808 /loss/FRt3/detection_loss‚1.6099172312962382 0loss/FRt4/assignment_loss‚2.4764669200648433 1loss/SRqqt1/detection_loss‚2.2885604338212446 /loss/FBt1/detection_loss‚2.2299548259803226 epoch‚9 .loss/FRt1/detection_loss‚2.798367515206337 0loss/FRt3/assignment_loss‚1.0861787691450955 2loss/SRqqt2/assignment_loss‚1.0637571472139593 /loss/FRt4/detection_loss‚2.2027645995435505 1loss/SRqqt2/detection_loss‚1.8958832583528884 1loss/SRqqt3/detection_loss‚0.8433719520637794 _runtime‚8907.302572247 trainer/global_step‚11399 .loss/FRt2/detection_loss‚2.361047171865586 2loss/SRqqt1/assignment_loss‚1.6993947394869544 1loss/FBt1/assignment_loss‚0.19132849843507366 1loss/SRqqt4/detection_loss‚1.7292304639609493 _step‚938 2loss/SRqqt4/assignment_loss‚1.2437706480981137 .loss/FBt3/detection_loss‚0.625582014504624 /loss/FBt4/detection_loss‚2.2279671331246695 %loss/total_loss‚38.29508485174885 0loss/FRt2/assignment_loss‚3.1308809765007184 /loss/FBt2/assignment_loss‚0.257564804164253 0loss/FBt3/assignment_loss‚0.0581060116850197 /loss/FBt2/detection_loss‚2.1346156232886844]À[c—)Ý 0loss/FBt3/assignment_loss‚0.0581060116850197 /loss/FBt2/detection_loss‚2.1346156232886844 % _timestamp‚1.7884138043148682e+09 0loss/FRt1/assignment_loss‚3.1201183259487153 2loss/SRqqt3/assignment_loss‚0.7803797069797987 1loss/FBt4/assignment_loss‚0.23987608101742808 /loss/FRt3/detection_loss‚1.6099172312962382 0loss/FRt4/assignment_loss‚2.4764669200648433 1loss/SRqqt1/detection_loss‚2.2885604338212446 /loss/FBt1/detection_loss‚2.2299548259803226 epoch‚9 .loss/FRt1/detection_loss‚2.798367515206337 0loss/FRt3/assignment_loss‚1.0861787691450955 2loss/SRqqt2/assignment_loss‚1.0637571472139593 /loss/FRt4/detection_loss‚2.2027645995435505 1loss/SRqqt2/detection_loss‚1.8958832583528884 1loss/SRqqt3/detection_loss‚0.8433719520637794 _runtime‚8907.302572247 trainer/global_step‚11399 .loss/FRt2/detection_loss‚2.361047171865586 2loss/SRqqt1/assignment_loss‚1.6993947394869544 1loss/FBt1/assignment_loss‚0.19132849843507366 1loss/SRqqt4/detection_loss‚1.7292304639609493 _step‚938 2loss/SRqqt4/assignment_loss‚1.2437706480981137 .loss/FBt3/detection_loss‚0.625582014504624 /loss/FBt4/detection_loss‚2.2279671331246695 %loss/total_loss‚38.29508485174885 0loss/FRt2/assignment_loss‚3.1308809765007184 /loss/FBt2/assignment_loss‚0.257564804164253ªNÈÅ’˜)‚(:‡ ïŽäÔ¨¶ý gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.124' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„¿-ñ…™)‚(:ú ïŽäÔÈ™ø disk.drbd2266.out‚0) proc.memory.availableMB‚ 3104.38671875! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚56264.39501953125 disk.rbd0.out‚0# memory_percent‚81.0523271560669 disk.sda.in‚ 0.1953125 disk.rbd1.in‚0 disk.rbd1.out‚36.36669921875 proc.cpu.threads‚17 network.recv‚ 3.271577e+06 disk.dm-0.in‚ 1187.62109375 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2810.9296875* proc.memory.percent‚17.156553268432617 cpu‚25.350814016172134 disk.sda1.in‚0( disk.nvme0c0n1.out‚56264.39501953125 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 network.sent‚ 2.8122149e+07* disk./.usagePercent‚16.669589078005863 disk.dm-1.in‚ 0.1953125 disk.sda3.out‚ 402.49609375 disk.sda.out‚ 402.49609375 disk.drbd2080.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.out‚0 disk.dm-1.out‚ 402.49609375# disk.dm-0.out‚56264.39501953125 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0% disk./.usageGB‚248.38892364501953 disk.drbd2265.out‚0Î&<³’š)‚(:‡ þŽäÔý©ˆ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.779' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#¬{óu›)‚(:ê þŽäÔõô“  disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0% disk.nvme0n1.out‚56330.9755859375 disk.drbd2265.in‚0 disk.drbd1716.out‚0 disk.sda1.in‚0 network.recv‚ 3.278602e+06* disk./.usagePercent‚16.669174601807928) proc.memory.availableMB‚ 3123.31640625 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.dm-1.out‚403.0625 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.drbd1716.in‚0$ memory_percent‚80.93678951263428" proc.memory.rssMB‚ 2810.9296875# disk.nvme0c0n1.in‚ 1187.62109375' disk.nvme0c0n1.out‚56330.9755859375 disk.rbd0.in‚0 network.sent‚ 2.823085e+07% disk./.usageGB‚248.38274765014648 disk.sda3.out‚403.0625 disk.drbd2080.out‚0 disk.dm-0.in‚ 1187.62109375" disk.dm-0.out‚56330.9755859375 disk.sda.out‚403.0625 cpu‚25.35030272552705 proc.cpu.threads‚17 disk.rbd1.out‚36.36669921875 disk.drbd2265.out‚0* proc.memory.percent‚17.156553268432617  Wwœ)‚(:’ äÔ¥Ÿÿ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚132.148' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.429599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QR׆)‚(:û äÔ…Ú¿  network.recv‚ 3.283482e+06# disk.dm-0.out‚56388.94677734375 disk.drbd2266.in‚0# disk.nvme0c0n1.in‚ 1187.62109375) proc.memory.availableMB‚ 3123.31640625 network.sent‚ 2.8237286e+07& disk.nvme0n1.out‚56388.94677734375 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda3.out‚ 403.40234375 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚36.36669921875! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 403.40234375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 1187.62109375 disk.sda.out‚ 403.40234375 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚17.156553268432617 cpu‚25.349820193935553 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.sda2.out‚0% disk./.usageGB‚248.41297149658203 disk.rbd0.out‚0( disk.nvme0c0n1.out‚56388.94677734375" proc.memory.rssMB‚ 2810.9296875 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚16.671202949501716 disk.drbd2080.in‚0$ memory_percent‚80.93678951263428öß2ž)- _wandbruntime‚8942  _runtime‚8942ÊÑùÊŸ)Ä _runtime‚8942.814051683 _step‚939 ' lr-AdamW/pg1‚2.3485736166075613e-06 ' lr-AdamW/pg2‚2.3485736166075613e-06 trainer/global_step‚11449 % _timestamp‚1.7884138398265824e+09ô4ÌÏ )É ' lr-AdamW/pg2‚2.3485736166075613e-06 trainer/global_step‚11449 % _timestamp‚1.7884138398265824e+09 _runtime‚8942.814051683 _step‚939 ' lr-AdamW/pg1‚2.3485736166075613e-06«\8ãÁ2¡)- _wandbruntime‚8942  _runtime‚8942ñ³·ìN¢)äO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97217070 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚737459762615808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚666141833625088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666141815622144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97217068 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.largÓÐS?œ$e_pool.allocated‚648679556969472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151970965 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚727397142460928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚675735168328192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚727397118520832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166617800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚648679539930112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216217974 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚11449 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166617798 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9592678340608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855812 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10062454433792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884138398273475e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚675735168328192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9592678340608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216238012 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9593334703104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚737459572954624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚940 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666141815622144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166617800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166617798 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9593334703104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚658236901063615 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216217974 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚675734493962752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9558015571259 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675734493962752 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10062620154880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚666141833625088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9592678340608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855812 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10062454433792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884138398273475e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚675735168328192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9592678340608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216238012 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9593334703104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚737459572954624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚940 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666141815622144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166617800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166617798 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9593334703104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚658236901063615 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216217974 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚675734493962752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9558015571259 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675734493962752 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10062620154880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚666141833625088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚648679539930112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216217974 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚11449 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756¬ª( 2¤)- _wandbruntime‚8943  _runtime‚8943ÔôÍJô&¥)îM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚658293293106463 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚11449 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658292621629139 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648734461100032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675792340808192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166631814 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382868242 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727458793198592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166631816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648734444060672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚666198168660992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675792340808192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9593497768448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8943.495953157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9558832006431 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216236428 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216256492 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151983712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9593497768448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675791666429440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666198186663936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166631814 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888308 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚727458769258496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10063476395520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382868242 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10063310687744 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚409 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9558177568467 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚737522269594112 % _timestamp‚1.7884138405077238e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216256492 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97225238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54758067 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888308 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚666198186663936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666198168660992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151983303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2±þb™, ¦)óM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚11449 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658292621629139 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648734461100032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675792340808192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166631814 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382868242 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727458793198592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166631816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648734444060672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚666198168660992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675792340808192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9593497768448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pXQ®7Íool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8943.495953157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9558832006431 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216236428 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216256492 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151983712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9593497768448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675791666429440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666198186663936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166631814 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888308 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚727458769258496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10063476395520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382868242 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10063310687744 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚409 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9558177568467 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚737522269594112 % _timestamp‚1.7884138405077238e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216256492 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97225238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54758067 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888308 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚666198186663936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666198168660992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151983303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚658293293106463 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560­dݱF2§)- _wandbruntime‚8943  _runtime‚8943hlÓ¶a¨)Û 1loss/SRqqt2/detection_loss‚1.6862605002365614 /loss/FBt3/detection_loss‚0.5986883938517217 %loss/total_loss‚39.54223784918872 0loss/SRqqt3/detection_loss‚1.029951858487147 % _timestamp‚1.7884138405098066e+09 /loss/FRt4/assignment_loss‚2.118959594280162 3loss/FBt4/assignment_loss‚0.0027332509751431646 /loss/FRt2/detection_loss‚2.1807285829863736 1loss/SRqqt1/detection_loss‚2.3936179247167377 /loss/FBt1/detection_loss‚1.9848790764808655 /loss/FBt4/detection_loss‚2.6778085112571715 0loss/FRt3/assignment_loss‚2.0339548751419665 1loss/FBt2/assignment_loss‚0.24131656524592213 epoch‚9 trainer/global_step‚11449 _runtime‚8943.497189841 2loss/SRqqt1/assignment_loss‚1.5872508105304506 2loss/SRqqt2/assignment_loss‚1.1170200646381105 2loss/SRqqt4/assignment_loss‚1.2477903474772454 /loss/FRt1/detection_loss‚2.9873003478973144 .loss/FRt3/detection_loss‚1.732681287562146 .loss/FRt4/detection_loss‚2.283751275945217 1loss/FBt1/assignment_loss‚0.14620693516917527 1loss/SRqqt4/detection_loss‚1.5832636699361622 0loss/FRt1/assignment_loss‚3.2902584806565316 /loss/FRt2/assignment_loss‚3.032861399153868 1loss/FBt3/assignment_loss‚0.04525913110909427 /loss/FBt2/detection_loss‚2.3297234007290433 2loss/SRqqt3/assignment_loss‚1.2099715647245965 _step‚942W¨cf©)à .loss/FRt4/detection_loss‚2.283751275945217 1loss/FBt1/assignment_loss‚0.14620693516917527 1loss/SRqqt4/detection_loss‚1.5832636699361622 0loss/FRt1/assignment_loss‚3.2902584806565316 /loss/FRt2/assignment_loss‚3.032861399153868 1loss/FBt3/assignment_loss‚0.04525913110909427 /loss/FBt2/detection_loss‚2.3297234007290433 2loss/SRqqt3/assignment_loss‚1.2099715647245965 _step‚942 1loss/SRqqt2/detection_loss‚1.6862605002365614 /loss/FBt3/detection_loss‚0.5986883938517217 %loss/total_loss‚39.54223784918872 0loss/SRqqt3/detection_loss‚1.029951858487147 % _timestamp‚1.7884138405098066e+09 /loss/FRt4/assignment_loss‚2.118959594280162 3loss/FBt4/assignment_loss‚0.0027332509751431646 /loss/FRt2/detection_loss‚2.1807285829863736 1loss/SRqqt1/detection_loss‚2.3936179247167377 /loss/FBt1/detection_loss‚1.9848790764808655 /loss/FBt4/detection_loss‚2.6778085112571715 0loss/FRt3/assignment_loss‚2.0339548751419665 1loss/FBt2/assignment_loss‚0.24131656524592213 epoch‚9 trainer/global_step‚11449 _runtime‚8943.497189841 2loss/SRqqt1/assignment_loss‚1.5872508105304506 2loss/SRqqt2/assignment_loss‚1.1170200646381105 2loss/SRqqt4/assignment_loss‚1.2477903474772454 /loss/FRt1/detection_loss‚2.9873003478973144 .loss/FRt3/detection_loss‚1.732681287562146®u\Ū)‚(:’ œäÔ˜Ùš gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.938' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.187599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¡}«)‚(:ò œäÔº–¦ & disk.nvme0n1.out‚56446.74267578125# disk.dm-0.out‚56446.74267578125# proc.memory.availableMB‚3123.25" proc.memory.rssMB‚ 2810.9296875( disk.nvme0c0n1.out‚56446.74267578125 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 network.sent‚ 2.8242458e+07 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 network.recv‚ 3.28745e+06 disk.sda2.out‚0 disk.rbd1.out‚36.36669921875 proc.cpu.threads‚17 disk.sda3.out‚ 403.99609375 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375 disk.sda.out‚ 403.99609375% disk./.usageGB‚248.42692947387695 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.out‚ 403.99609375$ memory_percent‚80.93719482421875* proc.memory.percent‚17.156553268432617( disk./.usagePercent‚16.6721396810695 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 cpu‚25.349276611071033£æïÁ’¬)‚(:‡ «äÔ쵉 gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚127.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0; ‚­)‚(:÷ «äÔ®éŽ  disk.sda2.out‚0 network.recv‚ 3.294264e+06* disk./.usagePercent‚16.673923797575622 disk.drbd2080.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 3122.87109375 disk.drbd2265.out‚0 disk.sda3.out‚ 404.3515625 disk.drbd2080.out‚0# disk.nvme0c0n1.in‚ 1187.62109375# disk.dm-0.out‚56512.02392578125 network.sent‚ 2.8351472e+07 disk.dm-1.out‚ 404.3515625( disk.nvme0c0n1.out‚56512.02392578125 disk.drbd1716.in‚0 cpu‚25.348772796280354 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚36.36669921875 proc.cpu.threads‚17$ disk./.usageGB‚248.4535140991211 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0! disk.nvme0n1.in‚ 1187.62109375$ memory_percent‚80.93950748443604 disk.dm-1.in‚ 0.1953125 disk.sda.out‚ 404.3515625 disk.sda3.in‚ 0.1953125& disk.nvme0n1.out‚56512.02392578125 disk.dm-0.in‚ 1187.62109375 disk.rbd0.out‚0 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2810.9296875* proc.memory.percent‚17.156553268432617 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda2.in‚0iý2®)- _wandbruntime‚8979  _runtime‚8979_UIʯ)Ä _step‚943 ' lr-AdamW/pg1‚1.9973263043349236e-06 ' lr-AdamW/pg2‚1.9973263043349236e-06 trainer/global_step‚11499 % _timestamp‚1.7884138763606577e+09 _runtime‚8979.348137779y(IÏ°)É ' lr-AdamW/pg1‚1.9973263043349236e-06 ' lr-AdamW/pg2‚1.9973263043349236e-06 trainer/global_step‚11499 % _timestamp‚1.7884138763606577e+09 _runtime‚8979.348137779 _step‚943¯ñwϬ2±)- _wandbruntime‚8979  _runtime‚8979 üé'²)ãO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678593806076416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678593131710976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9633664146432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661023614408139 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661022942931023 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 174981120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚730479655404032 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152606327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9634320508928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167318600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384482664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668959485567488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740585317078528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217164064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚11499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678593131710976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217144026 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9633664146432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚8979.349042126 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152606766 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9598196472911 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651424763497472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 384462624 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167318598ÂBã-Ä#³)èO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788413876361388e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167318598 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚668959467564544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97625470 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚668959485567488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97625468 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152606327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9634320508928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167318600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384482664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668959485567488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740585317078528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217164064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚11499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678593131710976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217144026 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9633664146432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚8979.349042126 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152606766 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9598196472911 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651424763497472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 384462624 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167318598 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678593806076416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678593131710976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9633664146432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661023614408139 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661022942931023 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 174981120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚730479655404032 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚945 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217162480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 # _timestamp‚1.78841387704974e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9599012907875 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669015820603392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97633636 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740647839137280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152619003 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10106343347712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97633638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167332614 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9634483574272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217182544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651479650588672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9634483574272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952„Zë†÷&¶)ñM _step‚945 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217162480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 # _timestamp‚1.78841387704974e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9599012907875 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669015820603392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97633636 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740647839137280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152619003 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10106343347712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97633638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167332614 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9634483574272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217182544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651479650588672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9634483574272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088±¨_s2·)- _wandbruntime‚8980  _runtime‚8980òn¯XY¸)Ó 0loss/FRt2/assignment_loss‚2.7445609671004276 2loss/SRqqt2/assignment_loss‚1.4337451398550045 /loss/FRt2/detection_loss‚1.9850930586919984 .loss/FBt1/detection_loss‚2.698734927177429 trainer/global_step‚11499 1loss/FBt2/assignment_loss‚0.21544193213698495 %loss/total_loss‚41.26036580780839 epoch‚9 .loss/FRt3/assignment_loss‚1.27013254924254 2loss/SRqqt3/assignment_loss‚0.7164306578242445 1loss/SRqqt4/detection_loss‚1.8725675350427629 /loss/FRt4/detection_loss‚2.5042488041978617 1loss/SRqqt3/detection_loss‚1.2206000493004405 /loss/FBt3/detection_loss‚0.5512836443303132 .loss/FBt4/detection_loss‚2.608205775419871 /loss/FBt2/detection_loss‚2.2802951633930206 _step‚946 1loss/SRqqt4/assignment_loss‚1.428306095711887 0loss/FBt1/assignment_loss‚1.5545874689700667 % _timestamp‚1.7884138770514684e+09 0loss/FRt4/assignment_loss‚2.7440628225986776 1loss/FBt3/assignment_loss‚0.10941871429034647 1loss/FBt4/assignment_loss‚0.07138493080856279 .loss/FRt1/detection_loss‚2.852706797917684 .loss/FRt3/detection_loss‚1.715758194367994 1loss/SRqqt1/detection_loss‚2.3429677061736585 _runtime‚8980.038862944 /loss/FRt1/assignment_loss‚2.676125212510427 1loss/SRqqt1/assignment_loss‚1.683043708372861 0loss/SRqqt2/detection_loss‚1.980663952373323+¹Ã`^¹)Ø epoch‚9 .loss/FRt3/assignment_loss‚1.27013254924254 2loss/SRqqt3/assignment_loss‚0.7164306578242445 1loss/SRqqt4/detection_loss‚1.8725675350427629 /loss/FRt4/detection_loss‚2.5042488041978617 1loss/SRqqt3/detection_loss‚1.2206000493004405 /loss/FBt3/detection_loss‚0.5512836443303132 .loss/FBt4/detection_loss‚2.608205775419871 /loss/FBt2/detection_loss‚2.2802951633930206 _step‚946 1loss/SRqqt4/assignment_loss‚1.428306095711887 0loss/FBt1/assignment_loss‚1.5545874689700667 % _timestamp‚1.7884138770514684e+09 0loss/FRt4/assignment_loss‚2.7440628225986776 1loss/FBt3/assignment_loss‚0.10941871429034647 1loss/FBt4/assignment_loss‚0.07138493080856279 .loss/FRt1/detection_loss‚2.852706797917684 .loss/FRt3/detection_loss‚1.715758194367994 1loss/SRqqt1/detection_loss‚2.3429677061736585 _runtime‚8980.038862944 /loss/FRt1/assignment_loss‚2.676125212510427 1loss/SRqqt1/assignment_loss‚1.683043708372861 0loss/SRqqt2/detection_loss‚1.980663952373323 0loss/FRt2/assignment_loss‚2.7445609671004276 2loss/SRqqt2/assignment_loss‚1.4337451398550045 /loss/FRt2/detection_loss‚1.9850930586919984 .loss/FBt1/detection_loss‚2.698734927177429 trainer/global_step‚11499 1loss/FBt2/assignment_loss‚0.21544193213698495 %loss/total_loss‚41.26036580780839²–öšx‘º)‚(:† ºäÔîé« gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.655' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.731 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mÓsÅx»)‚(:í ºäÔƒöÇ  disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda.out‚ 405.078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0$ memory_percent‚80.93972206115723" proc.memory.rssMB‚ 2810.9296875 disk.dm-1.in‚ 0.1953125 network.recv‚ 3.298328e+06 disk.sda3.out‚ 405.078125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚56577.072265625( proc.memory.availableMB‚ 3122.8359375! disk.nvme0n1.in‚ 1187.62109375 network.sent‚ 2.8357424e+07 disk.sda3.in‚ 0.1953125 cpu‚25.348244945459722 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 405.078125# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚36.45263671875 proc.cpu.threads‚17! disk.dm-0.out‚56577.072265625 disk.drbd2080.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.156553268432617) disk./.usagePercent‚16.67525554876874% disk./.usageGB‚248.47335815429688$ disk.nvme0n1.out‚56577.072265625w>]^¼)‚(:’ ÉäÔà´ˆ gpu.0.gpu‚25.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.561' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.312200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒc–…½)‚(:ú ÉäÔÓîù$ memory_percent‚80.93962669372559( proc.memory.availableMB‚ 3122.8515625* proc.memory.percent‚17.156553268432617 disk.rbd1.out‚36.51513671875 disk.sda3.out‚ 405.37109375 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2810.9296875 cpu‚25.347758725445463 network.sent‚ 2.8466558e+07% disk./.usageGB‚248.49143600463867! disk.nvme0n1.in‚ 1187.62109375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚56650.45361328125 disk.sda.in‚ 0.1953125 disk.sda1.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚56650.45361328125 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 network.recv‚ 3.305569e+06* disk./.usagePercent‚16.676468768473505 disk.sda2.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚56650.45361328125 disk.dm-1.out‚ 405.37109375 disk.drbd2265.out‚0 disk.sda.out‚ 405.37109375 disk.sda1.out‚0 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.drbd2266.out‚0ÕãÏ’¾)‚(:‡ ØäÔüƒˆ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.409' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€4ú…¿)‚(:ú ØäÔ‰¡ü  disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.8471943e+07 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.sda.in‚ 0.1953125! disk.nvme0n1.in‚ 1187.62109375# disk.dm-0.out‚56701.34033203125 disk.rbd0.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.rbd1.out‚36.51513671875$ disk./.usageGB‚248.5018081665039 disk.dm-1.in‚ 0.1953125 disk.drbd2266.out‚0$ memory_percent‚80.94122409820557) proc.memory.availableMB‚ 3122.58984375 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 network.recv‚ 3.309828e+06 disk.drbd2266.in‚0* proc.memory.percent‚17.156553268432617 cpu‚25.347280699736963 disk.dm-1.out‚ 406.00390625 disk.drbd2265.in‚0 disk.sda.out‚ 406.00390625 disk.sda3.out‚ 406.00390625 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2810.9296875* disk./.usagePercent‚16.677164852959102& disk.nvme0n1.out‚56701.34033203125 disk.dm-0.in‚ 1187.62109375( disk.nvme0c0n1.out‚56701.34033203125 disk.drbd1716.out‚0mfئ2À)- _wandbruntime‚9015  _runtime‚9015d¨q´ÈÁ) & lr-AdamW/pg1‚1.674323699110911e-06 & lr-AdamW/pg2‚1.674323699110911e-06 trainer/global_step‚11549 % _timestamp‚1.7884139128577447e+09 _runtime‚9015.845179586 _step‚947©„ZÍÂ)Ç _step‚947 & lr-AdamW/pg1‚1.674323699110911e-06 & lr-AdamW/pg2‚1.674323699110911e-06 trainer/global_step‚11549 % _timestamp‚1.7884139128577447e+09 _runtime‚9015.845179586³Aì2Ã)- _wandbruntime‚9015  _runtime‚9015ÂçæTÄ)äO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681452443824640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98033868 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚663809656275547 % _timestamp‚1.7884139128584318e+09 _runtime‚9015.846063634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218090116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98033870 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚743710777593856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚743710973546496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681451769459200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9675306314752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚663808984798431 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚733562216227328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 386109516 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 trainer/global_step‚11549 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9674649952256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218070078 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚671777137509888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733562192287232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10148585306624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088ž¤ï'Å)éO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10148757319168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚654169970025472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153241317 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218090116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9639031812319 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚671777119506944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168019400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55207449 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚681452443824640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9639686250075 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218070078 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 trainer/global_step‚11549 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9674649952256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218070078 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚671777137509888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733562192287232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10148585306624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681452443824640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98033868 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚663809656275547 % _timestamp‚1.7884139128584318e+09 _runtime‚9015.846063634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218090116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98033870 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚743710777593856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚743710973546496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681451769459200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9675306314752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚663808984798431 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚733562216227328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 386109516 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142012 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚11549 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 386121946 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚663864705363955 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚671833472545792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 % _timestamp‚1.7884139135506766e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚671833472545792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚949 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98042038 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚671833490548736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 386121946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168033416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153253958 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733623843024896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98042036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9640502685247 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚743773286775808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681508941925888 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚654224874156032 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9639848247283 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚681509616304640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚681509616304640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚733623866964992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10149607361536 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654224857116672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9675469380096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218088532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168033414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218108596 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚671833490548736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168033416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9676125755904 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218108596 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743773474326528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Z›©ý÷È)óM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681508941925888 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚654224874156032 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9639848247283 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚681509616304640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚681509616304640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚733623866964992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10149607361536 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654224857116672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9675469380096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218088532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168033414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218108596 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚671833490548736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168033416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9676125755904 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218108596 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743773474326528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 386142012 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168033414 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142012 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚11549 D6DeviceStatsMonitor.on_train_batch_end/ac¦Ü±ƒtive.all.freed‚ 386121946 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚663864705363955 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚671833472545792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 % _timestamp‚1.7884139135506766e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚671833472545792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚949 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98042038 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚671833490548736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 386121946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168033416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153253958 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733623843024896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98042036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9640502685247 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚743773286775808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844µ8Ǿ2É)- _wandbruntime‚9016  _runtime‚9016uþÌ>aÊ)Û 1loss/SRqqt2/detection_loss‚1.5510066412389278 1loss/FBt4/assignment_loss‚0.23776038766352162 1loss/SRqqt3/detection_loss‚1.1912717143165283 trainer/global_step‚11549 _runtime‚9016.539703101 0loss/FRt2/assignment_loss‚2.0825511309835645 0loss/FRt4/assignment_loss‚2.4389103966600754 2loss/SRqqt1/assignment_loss‚1.4958625478964103 2loss/FBt1/assignment_loss‚0.002678828706848435 1loss/SRqqt4/detection_loss‚1.4539312299478937 /loss/FBt2/detection_loss‚2.6129295006394386 % _timestamp‚1.7884139135523386e+09 /loss/FBt3/detection_loss‚0.5762277153486536 epoch‚9 0loss/FBt3/assignment_loss‚0.0976220772215987 /loss/FRt3/detection_loss‚1.5384696371108293 1loss/SRqqt1/detection_loss‚2.1980162435456325 .loss/FBt1/detection_loss‚2.691478304564953 %loss/total_loss‚37.90784743060402 0loss/FRt1/assignment_loss‚3.4151120255975163 1loss/SRqqt2/assignment_loss‚1.436514615903919 2loss/SRqqt3/assignment_loss‚0.9537013916176607 1loss/SRqqt4/assignment_loss‚1.010147405848666 1loss/FBt2/assignment_loss‚0.16360293889738386 /loss/FBt4/detection_loss‚2.4086733545575822 /loss/FRt2/detection_loss‚2.1275617460411205 /loss/FRt4/detection_loss‚2.0881824879085316 _step‚950 0loss/FRt3/assignment_loss‚1.3228697502613067 /loss/FRt1/detection_loss‚2.8127653581254624>÷°fË)à 0loss/FRt1/assignment_loss‚3.4151120255975163 1loss/SRqqt2/assignment_loss‚1.436514615903919 2loss/SRqqt3/assignment_loss‚0.9537013916176607 1loss/SRqqt4/assignment_loss‚1.010147405848666 1loss/FBt2/assignment_loss‚0.16360293889738386 /loss/FBt4/detection_loss‚2.4086733545575822 /loss/FRt2/detection_loss‚2.1275617460411205 /loss/FRt4/detection_loss‚2.0881824879085316 _step‚950 0loss/FRt3/assignment_loss‚1.3228697502613067 /loss/FRt1/detection_loss‚2.8127653581254624 1loss/SRqqt2/detection_loss‚1.5510066412389278 1loss/FBt4/assignment_loss‚0.23776038766352162 1loss/SRqqt3/detection_loss‚1.1912717143165283 trainer/global_step‚11549 _runtime‚9016.539703101 0loss/FRt2/assignment_loss‚2.0825511309835645 0loss/FRt4/assignment_loss‚2.4389103966600754 2loss/SRqqt1/assignment_loss‚1.4958625478964103 2loss/FBt1/assignment_loss‚0.002678828706848435 1loss/SRqqt4/detection_loss‚1.4539312299478937 /loss/FBt2/detection_loss‚2.6129295006394386 % _timestamp‚1.7884139135523386e+09 /loss/FBt3/detection_loss‚0.5762277153486536 epoch‚9 0loss/FBt3/assignment_loss‚0.0976220772215987 /loss/FRt3/detection_loss‚1.5384696371108293 1loss/SRqqt1/detection_loss‚2.1980162435456325 .loss/FBt1/detection_loss‚2.691478304564953 %loss/total_loss‚37.90784743060402¶@õ“Ë’Ì)‚(:‡ çäÔ¨Š§ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚130.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 FîiÍ)‚(:ô çäÔåì·  disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0! disk.nvme0n1.in‚ 1187.62109375 cpu‚25.346769392729275 network.sent‚ 2.8477972e+07 disk.drbd2265.out‚0 disk.sda3.out‚ 406.38671875 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚80.9401273727417 disk.dm-1.in‚ 0.1953125 disk.drbd2080.in‚0 disk.rbd1.out‚36.54736328125) proc.memory.availableMB‚ 3122.76953125* proc.memory.percent‚17.156553268432617 proc.cpu.threads‚17 disk.sda.out‚ 406.38671875& disk.nvme0c0n1.out‚56760.126953125 disk.dm-0.in‚ 1187.62109375 network.recv‚ 3.314226e+06 disk.dm-1.out‚ 406.38671875 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2810.9296875 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.drbd1716.out‚0! disk.dm-0.out‚56760.126953125 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375$ disk.nvme0n1.out‚56760.126953125 disk.drbd2266.out‚0* disk./.usagePercent‚16.677974604789906% disk./.usageGB‚248.51387405395508‚õmAœÎ)‚(:‘ öäÔŸ®ñ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.193' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.83860000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å[ñÏ)‚(:ô öäÔÈžÊ ! disk.nvme0n1.in‚ 1187.62109375$ memory_percent‚80.94160556793213 cpu‚25.346277838506662 disk.dm-1.in‚ 0.1953125 disk.drbd2080.in‚0& disk.nvme0c0n1.out‚56819.279296875" proc.memory.rssMB‚ 2810.9296875 disk.dm-1.out‚ 407.08203125 disk.drbd2266.in‚0 network.recv‚ 3.321491e+06 disk.sda.out‚ 407.08203125 disk.sda3.out‚ 407.08203125 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125) proc.memory.availableMB‚ 3122.52734375% disk./.usageGB‚248.53071975708008 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.8586623e+07! disk.dm-0.out‚56819.279296875 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚17.156553268432617 disk.sda.in‚ 0.1953125 disk.rbd0.in‚0$ disk.nvme0n1.out‚56819.279296875) disk./.usagePercent‚16.67910513406111 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚36.54736328125ŸHÒ72Ð)- _wandbruntime‚9052  _runtime‚9052¾ÔáäÇÑ)Á trainer/global_step‚11599 % _timestamp‚1.7884139498883305e+09 _runtime‚ 9052.87584307 _step‚951 & lr-AdamW/pg1‚1.379633114194355e-06 & lr-AdamW/pg2‚1.379633114194355e-06~{kpÌÒ)Æ & lr-AdamW/pg1‚1.379633114194355e-06 & lr-AdamW/pg2‚1.379633114194355e-06 trainer/global_step‚11599 % _timestamp‚1.7884139498883305e+09 _runtime‚ 9052.87584307 _step‚951·ë?K}2Ó)- _wandbruntime‚9052  _runtime‚9052Óû³Jè'Ô)âO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387736368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218996130 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736644729170432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168720198 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9715635758080 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153877730 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚684311081572864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153878171 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98442270 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55435901 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚746836349899264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10191782255616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666595026665839 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55435462 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736644753110528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168720200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚9052.876710607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚674594789452288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9680521589483 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9716292120576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9679867151727 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684310407207424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684310407207424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚684311081572864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218996130 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915159514112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98442268 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚674594771449344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10191620728832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚666595698142955 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387716328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚674594771449344}|U[í'Õ)çO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168720200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚9052.876710607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚674594789452288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9680521589483 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9716292120576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9679867151727 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684310407207424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684310407207424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚684311081572864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218996130 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915159514112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98442268 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚674594771449344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10191620728832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚666595698142955 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387716328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚674594771449344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387736368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218996130 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736644729170432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168720198 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9715635758080 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153877730 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚684311081572864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153878171 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98442270 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55435901 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚746836349899264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10191782255616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666595026665839 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55435462 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736644753110528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784¸§­€°2Ö)- _wandbruntime‚9053  _runtime‚9053Ïqá9k×)îM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98450436 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚684368254052864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚684367579674112 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55440396 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387748798 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153890383 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚11599 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674651124488192 _step‚953 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168734216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666650747231363 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚736706379908096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168734214 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219034648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚674651124488192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884139505864744e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10192637480448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168734216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153890834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚656970080684032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387748798 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219034648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55439947 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656970063644672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9681338024655 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674651142491136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430¯{Fù&Ø)óM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98450436 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚684368254052864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚684367579674112 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55440396 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387748798 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153890383 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚11599 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674651124488192 _step‚953 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168734216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666650747231363 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚736706379908096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168734214 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219034648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚674651124488192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884139505864744e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10192637480448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168734216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153890834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚656970080684032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387748798 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219034648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55439947 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656970063644672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9681338024655 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674651142491136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430¹B 2Ù)- _wandbruntime‚9053  _runtime‚9053J(—`Ú)Ú 2loss/SRqqt4/assignment_loss‚1.2188821017140679 trainer/global_step‚11599 1loss/SRqqt1/assignment_loss‚1.999983035493642 1loss/SRqqt2/assignment_loss‚1.197525143115358 1loss/SRqqt3/detection_loss‚0.8819064934768023 0loss/FRt2/assignment_loss‚2.7329138852655888 0loss/FRt4/assignment_loss‚2.5148187189731956 1loss/FBt2/assignment_loss‚0.37641651777084917 /loss/FRt3/detection_loss‚1.3341570865166814 /loss/FBt4/detection_loss‚3.0511505603790283 &loss/total_loss‚41.738870141878394 0loss/FRt3/assignment_loss‚1.2788200618927938 1loss/FBt1/assignment_loss‚0.12584562498765686 1loss/SRqqt2/detection_loss‚1.6337116531350395 % _timestamp‚1.7884139505885906e+09 .loss/FRt4/detection_loss‚2.250524326315466 1loss/SRqqt1/detection_loss‚2.3384895706549287 /loss/FBt2/detection_loss‚2.8169879093766212 2loss/SRqqt3/assignment_loss‚0.8157114909310269 .loss/FBt1/detection_loss‚3.536044657230377 _runtime‚9053.575966391 1loss/FBt4/assignment_loss‚0.13347571039776085 /loss/FRt2/detection_loss‚2.0551389286092308 1loss/SRqqt4/detection_loss‚1.9345028532358508 epoch‚9 _step‚954 0loss/FRt1/assignment_loss‚3.8004162002492836 1loss/FBt3/assignment_loss‚0.06512727800079367 .loss/FRt1/detection_loss‚3.111037477299019 /loss/FBt3/detection_loss‚0.5352828568573316þ>øeÛ)ß 1loss/SRqqt4/detection_loss‚1.9345028532358508 epoch‚9 _step‚954 0loss/FRt1/assignment_loss‚3.8004162002492836 1loss/FBt3/assignment_loss‚0.06512727800079367 .loss/FRt1/detection_loss‚3.111037477299019 /loss/FBt3/detection_loss‚0.5352828568573316 2loss/SRqqt4/assignment_loss‚1.2188821017140679 trainer/global_step‚11599 1loss/SRqqt1/assignment_loss‚1.999983035493642 1loss/SRqqt2/assignment_loss‚1.197525143115358 1loss/SRqqt3/detection_loss‚0.8819064934768023 0loss/FRt2/assignment_loss‚2.7329138852655888 0loss/FRt4/assignment_loss‚2.5148187189731956 1loss/FBt2/assignment_loss‚0.37641651777084917 /loss/FRt3/detection_loss‚1.3341570865166814 /loss/FBt4/detection_loss‚3.0511505603790283 &loss/total_loss‚41.738870141878394 0loss/FRt3/assignment_loss‚1.2788200618927938 1loss/FBt1/assignment_loss‚0.12584562498765686 1loss/SRqqt2/detection_loss‚1.6337116531350395 % _timestamp‚1.7884139505885906e+09 .loss/FRt4/detection_loss‚2.250524326315466 1loss/SRqqt1/detection_loss‚2.3384895706549287 /loss/FBt2/detection_loss‚2.8169879093766212 2loss/SRqqt3/assignment_loss‚0.8157114909310269 .loss/FBt1/detection_loss‚3.536044657230377 _runtime‚9053.575966391 1loss/FBt4/assignment_loss‚0.13347571039776085 /loss/FRt2/detection_loss‚2.0551389286092308ºb‘95’Ü)‚(:‡ …äÔÃñ¶ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.734' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rtÏЀÝ)‚(:õ …äÔÁḠ disk.sda.in‚ 0.1953125* disk./.usagePercent‚16.678227028218668 disk.drbd1716.out‚0 disk.dm-0.in‚ 1187.62109375 disk.dm-1.out‚ 407.63671875* proc.memory.percent‚17.156553268432617 cpu‚25.34578935420584 disk.sda3.out‚ 407.63671875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2810.9296875! disk.nvme0n1.in‚ 1187.62109375 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda.out‚ 407.63671875 disk.drbd2266.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚56903.0712890625 disk.rbd1.in‚0% disk.nvme0n1.out‚56903.0712890625) proc.memory.availableMB‚ 3122.73828125 disk.rbd1.out‚ 36.6103515625 network.sent‚ 2.8593029e+07 network.recv‚ 3.32608e+06% disk./.usageGB‚248.51763534545898 proc.cpu.threads‚17 disk.drbd1716.in‚0" disk.dm-0.out‚56903.0712890625 disk.sda3.in‚ 0.1953125 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0$ memory_percent‚80.94031810760498 disk.rbd0.in‚0=¨ø“Þ)‚(:’ ”äÔ¹ÓŠ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06W–Ó|ß)‚(:ñ ”äÔüö†  disk.drbd2266.out‚0 disk.rbd1.in‚0$ memory_percent‚80.94034194946289 network.recv‚ 3.332839e+06 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.sda3.out‚ 408.1796875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚56940.4384765625 network.sent‚ 2.8701213e+07* disk./.usagePercent‚16.680035721448075 disk.drbd1716.in‚0" disk.dm-0.out‚56940.4384765625' proc.memory.availableMB‚ 3122.734375* proc.memory.percent‚17.156553268432617 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375' disk.nvme0c0n1.out‚56940.4384765625 disk.drbd2265.out‚0 disk.sda.out‚ 408.1796875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1187.62109375" proc.memory.rssMB‚ 2810.9296875 cpu‚25.34529824183068 disk.dm-1.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 36.6103515625 proc.cpu.threads‚17% disk./.usageGB‚248.54458618164062 disk.dm-1.out‚ 408.1796875 disk.drbd2080.out‚0! disk.nvme0n1.in‚ 1187.62109375·˜¼ý2à)- _wandbruntime‚9089  _runtime‚9089°?ÃXÊá)Ä _step‚955 ' lr-AdamW/pg1‚1.1133159626614214e-06 ' lr-AdamW/pg2‚1.1133159626614214e-06 trainer/global_step‚11649 % _timestamp‚1.7884139867639337e+09 _runtime‚9089.751393044g¥£Ïâ)É ' lr-AdamW/pg2‚1.1133159626614214e-06 trainer/global_step‚11649 % _timestamp‚1.7884139867639337e+09 _runtime‚9089.751393044 _step‚955 ' lr-AdamW/pg1‚1.1133159626614214e-06»Ãï%·2ã)- _wandbruntime‚9089  _runtime‚9089=ăîýä)äO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10234544681984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9757277926400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389363220 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚749961810735616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98850668 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739727289993728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚659660383081472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169420998 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659660366042112 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219942220 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219922182 trainer/global_step‚11649 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677412441394688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9757277926400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚687169044955648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55663954 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚687169719321088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚669381740010363 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚9089.752297962 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687169044955648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169420998 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154514624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55663505 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚739727266053632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154514173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98850670 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389343180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677412423391744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749962000396800 % _timestamp‚1.7884139867646146e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169420998 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659660366042112 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219942220 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219922182 trainer/global_step‚11649 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677412441394688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9757277926400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚687169044955648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55663954 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚687169719321088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚669381740010363 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚9089.752297962 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687169044955648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169420998 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154514624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55663505 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚739727266053632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154514173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98850670 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389343180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677412423391744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749962000396800 % _timestamp‚1.7884139867646146e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395716 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚677468794433536 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154526905 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169435016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55668069 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9721518926099 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9758097367552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98858836 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10235575828992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659715270172672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750024516560384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9757440991744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚687226217422336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10235416412672 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚677468794433536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128­Ã]¨¿!ê)òM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚677468776430592 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154527343 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚739788916791296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚9090.465378965 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9722173364063 $ _timestamp‚1.788413987477534e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9757440991744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚750024333203968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚957 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750024516560384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9757440991744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚687226217422336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10235416412672 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚677468794433536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389395716 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98858838 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169435016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚659715287212032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169435014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677468776430592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389375650 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395716 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚677468794433536 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154526905 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169435016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55668069 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9721518926099 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9758097367552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98858836 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10235575828992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659715270172672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚672167110400655 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9797607369728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680230075334144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690027682703872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170121798 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10277552400384 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690028357069312 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9798263732224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99259068 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55890658 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220848234 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170121798 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚662405572570112 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155149728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚742809826876928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55890201 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690027682703872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155149269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚680230093337088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220848234 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚753087193810432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99259068 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55890658 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220848234 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170121798 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚662405572570112 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155149728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚742809826876928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55890201 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690027682703872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155149269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚680230093337088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220848234 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚753087193810432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚672167110400655 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9797607369728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680230075334144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690027682703872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170121798 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10277552400384 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414À¡—2ø)- _wandbruntime‚9127  _runtime‚9127îìÏr<ù)íM _step‚961 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 391002502 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170135816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680286446375936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788414024727233e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155162413 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155161960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10278246291968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672223502443503 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9798426797568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55894724 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680286428372992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220886752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99267236 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9799083173376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10278409902592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9799083173376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391022568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9798426797568 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220886752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚11699 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚742871453674496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9762354265507 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170135814 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672222830966179 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220866688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚680286446375936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99267238 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚753149887517184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220866688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚680286446375936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99267238 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚753149887517184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788414024727233e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155162413 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155161960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10278246291968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672223502443503 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9798426797568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55894724 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680286428372992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220886752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99267236 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9799083173376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10278409902592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9799083173376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908Á¶Ö]2û)- _wandbruntime‚9127  _runtime‚9127 1ô]ü)× 2loss/SRqqt4/assignment_loss‚1.5299906114764188 .loss/FRt4/detection_loss‚2.544369052997545 1loss/SRqqt4/detection_loss‚1.9395429647487143 % _timestamp‚1.7884140247291176e+09 0loss/FRt1/assignment_loss‚3.0275796972788296 1loss/FBt4/assignment_loss‚0.13817901725630577 %loss/total_loss‚40.82880366911691 epoch‚9 _step‚962 0loss/FRt2/assignment_loss‚2.4995372877234505 .loss/FRt1/detection_loss‚3.120006651832507 1loss/SRqqt2/detection_loss‚1.7331583932042123 trainer/global_step‚11699 1loss/FBt1/assignment_loss‚0.08791909655297303 1loss/FBt3/assignment_loss‚0.22429612719789904 1loss/SRqqt3/detection_loss‚0.7745053890819956 /loss/FBt2/detection_loss‚2.4582918703556063 _runtime‚9127.716486342 0loss/FRt3/assignment_loss‚1.6599143269110699 /loss/FRt4/assignment_loss‚2.399049708717748 2loss/SRqqt1/assignment_loss‚1.6285834538214135 2loss/SRqqt2/assignment_loss‚1.1675755110653965 1loss/FBt2/assignment_loss‚0.09656125362962484 .loss/FRt2/detection_loss‚2.313217207513782 .loss/FRt3/detection_loss‚2.014736226531772 .loss/FBt1/detection_loss‚2.735045380062527 1loss/SRqqt1/detection_loss‚2.3792052603129186 /loss/FBt3/detection_loss‚0.5504218897767714 /loss/FBt4/detection_loss‚2.6788162775337696 1loss/SRqqt3/assignment_loss‚1.128301013533662ôöIþbý)Ü 1loss/SRqqt3/assignment_loss‚1.128301013533662 2loss/SRqqt4/assignment_loss‚1.5299906114764188 .loss/FRt4/detection_loss‚2.544369052997545 1loss/SRqqt4/detection_loss‚1.9395429647487143 % _timestamp‚1.7884140247291176e+09 0loss/FRt1/assignment_loss‚3.0275796972788296 1loss/FBt4/assignment_loss‚0.13817901725630577 %loss/total_loss‚40.82880366911691 epoch‚9 _step‚962 0loss/FRt2/assignment_loss‚2.4995372877234505 .loss/FRt1/detection_loss‚3.120006651832507 1loss/SRqqt2/detection_loss‚1.7331583932042123 trainer/global_step‚11699 1loss/FBt1/assignment_loss‚0.08791909655297303 1loss/FBt3/assignment_loss‚0.22429612719789904 1loss/SRqqt3/detection_loss‚0.7745053890819956 /loss/FBt2/detection_loss‚2.4582918703556063 _runtime‚9127.716486342 0loss/FRt3/assignment_loss‚1.6599143269110699 /loss/FRt4/assignment_loss‚2.399049708717748 2loss/SRqqt1/assignment_loss‚1.6285834538214135 2loss/SRqqt2/assignment_loss‚1.1675755110653965 1loss/FBt2/assignment_loss‚0.09656125362962484 .loss/FRt2/detection_loss‚2.313217207513782 .loss/FRt3/detection_loss‚2.014736226531772 .loss/FBt1/detection_loss‚2.735045380062527 1loss/SRqqt1/detection_loss‚2.3792052603129186 /loss/FBt3/detection_loss‚0.5504218897767714 /loss/FBt4/detection_loss‚2.6788162775337696ÂÏ\B‘þ)‚(:† ÐäÔ¾Á gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.903 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04°|uÿ)‚(:ê ÐäÔêîþ * disk./.usagePercent‚16.685621805845198$ disk.nvme0n1.out‚57191.029296875 disk.dm-1.out‚ 410.21875 disk.rbd1.out‚36.67333984375) proc.memory.availableMB‚ 3208.92578125 cpu‚25.343337176144576 network.sent‚ 2.8907449e+07% disk./.usageGB‚248.62782287597656! disk.dm-0.out‚57191.029296875 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚57191.029296875# proc.memory.rssMB‚ 2724.73046875( proc.memory.percent‚16.6304349899292 disk.sda.in‚ 0.1953125 disk.sda.out‚ 410.21875 disk.sda3.out‚ 410.21875 disk.drbd2266.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0# memory_percent‚80.4142713546753 network.recv‚ 3.353442e+06 disk.drbd2266.in‚0yÍIf€*‚(:’ ßäÔåò• gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.631' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.126200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èŽÅ*{*‚(:ð ßäÔ´–Ô  disk.sda1.out‚0 disk.drbd1716.out‚0! disk.dm-0.out‚57279.974609375 disk.sda3.out‚ 410.58203125 disk.drbd2266.out‚0& disk.nvme0c0n1.out‚57279.974609375 disk.dm-0.in‚ 1187.62109375 disk.drbd2265.in‚0 network.sent‚ 2.9016223e+07 disk.rbd0.out‚0 disk.rbd1.out‚36.67333984375 disk.dm-1.out‚ 410.58203125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0$ memory_percent‚80.41400909423828 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1187.62109375& proc.memory.availableMB‚ 3208.96875 disk.sda.in‚ 0.1953125* disk./.usagePercent‚16.687446115534193$ disk./.usageGB‚248.6550064086914 disk.drbd2080.out‚0 disk.drbd1716.in‚0$ disk.nvme0n1.out‚57279.974609375 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.sda.out‚ 410.58203125# proc.memory.rssMB‚ 2724.73046875( proc.memory.percent‚16.6304349899292 cpu‚25.342859423052666 network.recv‚ 3.360114e+06 disk.drbd2080.in‚0áe‹2‚*- _wandbruntime‚9164  _runtime‚9164…ÒVDǃ*Á trainer/global_step‚11749 % _timestamp‚1.7884140612336557e+09 _runtime‚ 9164.22110578 _step‚963 & lr-AdamW/pg1‚6.660180355801622e-07 & lr-AdamW/pg2‚6.660180355801622e-07_ê9Ì„*Æ & lr-AdamW/pg1‚6.660180355801622e-07 & lr-AdamW/pg2‚6.660180355801622e-07 trainer/global_step‚11749 % _timestamp‚1.7884140612336557e+09 _runtime‚ 9164.22110578 _step‚963äá2…*- _wandbruntime‚9164  _runtime‚9164¿²Uê'†*äO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚745892363760128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚665150796137472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9838593175552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚11749 _runtime‚9164.222108362 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚964 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221774286 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221794324 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674953152268063 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683047745279488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170822598 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚683047727276544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674953823745179 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221794324 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756212862132736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9802373169951 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683047745279488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9839249538048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 392616924 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56119222 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170822600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9803027607707 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692886320452096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚745892339820032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10320688033792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665150779098112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170822600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155786692 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683047727276544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155786240 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚692886320452096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221794324 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674953152268063 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683047745279488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 6ªwk ÷%56362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170822598 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚683047727276544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674953823745179 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221794324 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756212862132736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9802373169951 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683047745279488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9839249538048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 392616924 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56119222 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170822600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9803027607707 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692886320452096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚745892339820032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10320688033792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665150779098112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170822600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155786692 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683047727276544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155786240 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚692886320452096 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683104080315392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392649420 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170836614 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9840068979200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9803844042879 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683104098318336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649420 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170836616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99675636 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692944167297536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9840068979200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚756275385290240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675009544310911 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155798946 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99675638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683104080315392 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665205683228672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9839412603392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10321394732544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9803189604915 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170836616 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221792740 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 392629354 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221812804 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788414061930177e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚745953990557696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚692944167297536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56123772 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629354 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 9164.91802242 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170836614 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155799410 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚117493·c³÷&Š*ñM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚745954014497792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚665205700268032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649420 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170836616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99675636 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692944167297536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9840068979200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚756275385290240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675009544310911 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155798946 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99675638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683104080315392 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665205683228672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9839412603392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10321394732544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9803189604915 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170836616 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221792740 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 392629354 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221812804 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788414061930177e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚745953990557696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚692944167297536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56123772 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629354 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 9164.91802242 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170836614 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155799410 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚11749 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683104080315392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392649420 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170836614 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9840068979200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9803844042879 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683104098318336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695744958200320 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222720376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚667896002665472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9843208509359 trainer/global_step‚11799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚759338617437184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748974900643328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171523398 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56347258 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677739865612587 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _step‚968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 394223736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚685865379218944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685865397221888 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚677739194135471 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9843862947115 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚695745632565760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748974876703232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171523400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222700338 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171523400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695744958200320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9879578981376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222700338 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222720376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚695745632565760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748974876703232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171523400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222700338 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171523400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695744958200320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9879578981376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222700338 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222720376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695744958200320 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222720376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚667896002665472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9843208509359 trainer/global_step‚11799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚759338617437184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748974900643328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171523398 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56347258 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677739865612587 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _step‚968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467ÈKAjñ2š*- _wandbruntime‚9201  _runtime‚9201Ál½K!›*ïM _runtime‚9201.445874422 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56352256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156435827 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 394256206 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚467 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100084038 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156436294 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚759401105832448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10364378257920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9881054785024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667950889756672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100084036 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚685921732257792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171537416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685921750260736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9881054785024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171537414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10364554451456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685921750260736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚969 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171537414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚749036551380992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9844679382287 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695802805045760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDevi egÔ ceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9844024944323 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222738856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222738856 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222718792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚695802130667008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222718792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677795586178319 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚759400905698816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚695802805045760Éœ=:ú&œ*ôM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9844024944323 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222738856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222738856 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222718792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚695802130667008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222718792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677795586178319 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚759400905698816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚695802805045760 _runtime‚9201.445874422 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56352256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156435827 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 394256206 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚467 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100084038 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156436294 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚759401105832448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10364378257920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9881054785024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667950889756672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100084036 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚685921732257792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171537416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685921750260736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9881054785024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171537414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10364554451456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685921750260736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚969 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171537414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚749036551380992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9844679382287 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695802805045760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F]ž*× 0loss/FRt4/assignment_loss‚2.1402152402456416 1loss/SRqqt3/assignment_loss‚0.870580138761158 1loss/FBt4/assignment_loss‚0.06310737411937832 1loss/SRqqt2/detection_loss‚1.7995661021769047 .loss/FBt3/detection_loss‚0.638449233093792 epoch‚9 2loss/SRqqt2/assignment_loss‚1.2831658942997455 -loss/FRt2/detection_loss‚2.11926612698219 /loss/FRt4/detection_loss‚2.4192387735427814 1loss/SRqqt3/detection_loss‚0.8257385456052265 /loss/FBt4/detection_loss‚2.4319571256637573 % _timestamp‚1.7884140984594953e+09 0loss/FRt1/assignment_loss‚2.6025980847222465 /loss/FRt1/detection_loss‚2.7878509759902954 /loss/FRt3/detection_loss‚1.6529664315835193 0loss/SRqqt4/detection_loss‚1.658304367305302 trainer/global_step‚11799 /loss/FRt2/assignment_loss‚2.870527572252534 0loss/FRt3/assignment_loss‚1.2293336832964863 2loss/SRqqt1/assignment_loss‚1.5817060828208924 2loss/FBt2/assignment_loss‚0.023071109782904387 .loss/FBt2/detection_loss‚4.082105822861195 2loss/SRqqt4/assignment_loss‚1.4040471627944853 0loss/FBt1/assignment_loss‚0.1902109032186369 1loss/SRqqt1/detection_loss‚2.4349960599626814 1loss/FBt3/assignment_loss‚0.16251888551714727 /loss/FBt1/detection_loss‚2.7686321669154697 %loss/total_loss‚40.04015386351437 _runtime‚9201.446862967 _step‚970¥Í_ bŸ*Ü 0loss/FRt1/assignment_loss‚2.6025980847222465 /loss/FRt1/detection_loss‚2.7878509759902954 /loss/FRt3/detection_loss‚1.6529664315835193 0loss/SRqqt4/detection_loss‚1.658304367305302 trainer/global_step‚11799 /loss/FRt2/assignment_loss‚2.870527572252534 0loss/FRt3/assignment_loss‚1.2293336832964863 2loss/SRqqt1/assignment_loss‚1.5817060828208924 2loss/FBt2/assignment_loss‚0.023071109782904387 .loss/FBt2/detection_loss‚4.082105822861195 2loss/SRqqt4/assignment_loss‚1.4040471627944853 0loss/FBt1/assignment_loss‚0.1902109032186369 1loss/SRqqt1/detection_loss‚2.4349960599626814 1loss/FBt3/assignment_loss‚0.16251888551714727 /loss/FBt1/detection_loss‚2.7686321669154697 %loss/total_loss‚40.04015386351437 _runtime‚9201.446862967 _step‚970 0loss/FRt4/assignment_loss‚2.1402152402456416 1loss/SRqqt3/assignment_loss‚0.870580138761158 1loss/FBt4/assignment_loss‚0.06310737411937832 1loss/SRqqt2/detection_loss‚1.7995661021769047 .loss/FBt3/detection_loss‚0.638449233093792 epoch‚9 2loss/SRqqt2/assignment_loss‚1.2831658942997455 -loss/FRt2/detection_loss‚2.11926612698219 /loss/FRt4/detection_loss‚2.4192387735427814 1loss/SRqqt3/detection_loss‚0.8257385456052265 /loss/FBt4/detection_loss‚2.4319571256637573 % _timestamp‚1.7884140984594953e+09Ê^y¦ *‚(:’ ›‘äÔÒßñ gpu.0.gpu‚25.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.595800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C…¡*‚(:ô ›‘äÔ¸»ó disk.drbd2266.out‚0 cpu‚25.340933977578885 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚57541.8271484375) proc.memory.availableMB‚ 3208.50390625 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.sda.in‚ 0.1953125$ memory_percent‚80.41684627532959# proc.memory.rssMB‚ 2724.73046875 proc.cpu.threads‚17 network.sent‚ 2.9142566e+07* disk./.usagePercent‚16.692478711642288 disk.rbd1.in‚0 disk.sda.out‚ 412.5859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.1953125 network.recv‚ 3.379831e+06 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚36.67333984375 disk.dm-1.out‚ 412.5859375% disk./.usageGB‚248.72999572753906 disk.dm-0.in‚ 1187.62109375" disk.dm-0.out‚57541.8271484375 disk.sda3.in‚ 0.1953125 disk.sda3.out‚ 412.5859375( proc.memory.percent‚16.6304349899292 disk.sda1.out‚0% disk.nvme0n1.out‚57541.8271484375 disk.drbd2265.in‚0 disk.drbd2265.out‚0w é’¢*‚(:‡ ª‘äÔû£µ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.761' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çoõv£*‚(:ë ª‘äÔ÷Œª  network.sent‚ 2.9251389e+07% disk./.usageGB‚248.73362350463867 disk.rbd0.out‚0 disk.sda2.in‚0% disk.nvme0n1.out‚57595.4326171875" disk.dm-0.out‚57595.4326171875 disk.sda.in‚ 0.1953125 disk.sda.out‚ 413.5390625" proc.memory.availableMB‚3208.5# proc.memory.rssMB‚ 2724.73046875 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1187.62109375' disk.nvme0c0n1.out‚57595.4326171875 disk.drbd2265.out‚0 disk.sda3.out‚ 413.5390625 disk.drbd2266.in‚0( proc.memory.percent‚16.6304349899292 disk.sda1.out‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 network.recv‚ 3.386708e+06) disk./.usagePercent‚16.69272217480735 disk.rbd0.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.dm-1.out‚ 413.5390625 disk.drbd2080.out‚0 disk.dm-0.in‚ 1187.62109375 disk.sda3.in‚ 0.1953125 disk.drbd1716.out‚0 disk.sda1.in‚0 disk.drbd2080.in‚0# memory_percent‚80.4168701171875 disk.rbd1.in‚0 disk.rbd1.out‚36.67333984375 disk.drbd2265.in‚0 cpu‚25.340448369574275»I}Š2¤*- _wandbruntime‚9237  _runtime‚9237hïÊ¥*Ä ' lr-AdamW/pg1‚3.3280276331137654e-07 ' lr-AdamW/pg2‚3.3280276331137654e-07 trainer/global_step‚11849 % _timestamp‚1.7884141348014443e+09 _runtime‚9237.788928135 _step‚971•¥ÞìϦ*É _step‚971 ' lr-AdamW/pg1‚3.3280276331137654e-07 ' lr-AdamW/pg2‚3.3280276331137654e-07 trainer/global_step‚11849 % _timestamp‚1.7884141348014443e+09 _runtime‚9237.788928135ËÏ>äÀ2§*- _wandbruntime‚9237  _runtime‚9237"¦ì'¨*æO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚688683031161344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223626390 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚680525236002879 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚688683031161344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56575873 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚9237.789743533 _step‚972 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172224198 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172224200 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100484268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚762464228677632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚680525907479995 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670641209193472 trainer/global_step‚11849 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223646428 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10406791151104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9920564787200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223626390 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157060141 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10406629624320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9921221149696 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56576312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚752057437526528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395850588 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698604270313984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9884043848767 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223646428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9884698286523 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚688683049164288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688683049164288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884141348021398e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698603595948544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9921221149696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚698604270313984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Ëœü»Ë ©*ëO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚688683049164288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688683049164288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884141348021398e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698603595948544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9921221149696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚698604270313984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚688683031161344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223626390 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚680525236002879 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚688683031161344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56575873 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚9237.789743533 _step‚972 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172224198 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak&ØÉ&‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172224200 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100484268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚762464228677632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚680525907479995 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670641209193472 trainer/global_step‚11849 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223646428 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10406791151104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9920564787200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223626390 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157060141 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10406629624320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9921221149696 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56576312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚752057437526528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395850588 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698604270313984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9884043848767 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223646428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9884698286523 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚752119064324096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395883058 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10407480236032 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172238214 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚680580956568403 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9921384215040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚688739402203136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚698661442793984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚688739384200192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9922040590848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56580442 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157072878 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172238214 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172238216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100492436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚762526544560128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698660768415232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752119088264192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223664908 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚762526740499456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670696096284672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9885514721695 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395883058 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9884860283731 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903124 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100492438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚698661442793984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172238216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚680581628045727 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚11849 % _timestamp‚1.7884141354873762e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56580892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670696113324032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223644844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223644844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚688739384200192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903124 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9884860283731 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903124 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100492438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚698661442793984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172238216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚680581628045727 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚11849 % _timestamp‚1.7884141354873762e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56580892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670696113324032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223644844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223644844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚688739384200192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903124 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚752119064324096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395883058 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10407480236032 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172238214 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚680580956568403 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9921384215040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚688739402203136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚698661442793984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚688739384200192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9922040590848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56580442 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157072878 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172238214 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172238216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100492436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚762526544560128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698660768415232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752119088264192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223664908 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚762526740499456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670696096284672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9885514721695 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928ÍfâH2­*- _wandbruntime‚9238  _runtime‚9238{tŠ…X®*Ò 1loss/SRqqt2/assignment_loss‚1.280392823291236 .loss/FRt2/detection_loss‚2.513989941088593 1loss/SRqqt1/assignment_loss‚1.736032762174164 epoch‚9 /loss/FRt2/assignment_loss‚3.238131770962163 2loss/SRqqt3/assignment_loss‚0.8318028625227132 /loss/FRt3/detection_loss‚1.6361391195454276 1loss/SRqqt2/detection_loss‚1.4360269681903823 1loss/SRqqt4/detection_loss‚1.9043644318977992 /loss/FBt3/detection_loss‚0.5647941326333947 % _timestamp‚1.7884141354888878e+09 _runtime‚9238.476215807 %loss/total_loss‚42.84245321107451 /loss/FRt1/assignment_loss‚3.607986974219481 /loss/FRt4/assignment_loss‚3.190906022489071 1loss/FBt3/assignment_loss‚0.12905692147595954 2loss/FBt4/assignment_loss‚0.006057147276123942 .loss/FRt1/detection_loss‚3.312988266348839 .loss/FRt4/detection_loss‚2.418524920940399 /loss/FBt1/detection_loss‚2.3848951935768126 -loss/FBt4/detection_loss‚2.44932429989179 trainer/global_step‚11849 2loss/SRqqt4/assignment_loss‚1.4058446781916751 1loss/FBt2/assignment_loss‚0.30086570025378023 0loss/SRqqt1/detection_loss‚2.455374235107053 _step‚974 /loss/FRt3/assignment_loss‚1.409757681764089 1loss/FBt1/assignment_loss‚0.12867408321471813 1loss/SRqqt3/detection_loss‚1.1215157243028535 .loss/FBt2/detection_loss‚3.379006549715996XëÎ]¯*× 1loss/SRqqt2/detection_loss‚1.4360269681903823 1loss/SRqqt4/detection_loss‚1.9043644318977992 /loss/FBt3/detection_loss‚0.5647941326333947 % _timestamp‚1.7884141354888878e+09 _runtime‚9238.476215807 %loss/total_loss‚42.84245321107451 /loss/FRt1/assignment_loss‚3.607986974219481 /loss/FRt4/assignment_loss‚3.190906022489071 1loss/FBt3/assignment_loss‚0.12905692147595954 2loss/FBt4/assignment_loss‚0.006057147276123942 .loss/FRt1/detection_loss‚3.312988266348839 .loss/FRt4/detection_loss‚2.418524920940399 /loss/FBt1/detection_loss‚2.3848951935768126 -loss/FBt4/detection_loss‚2.44932429989179 trainer/global_step‚11849 2loss/SRqqt4/assignment_loss‚1.4058446781916751 1loss/FBt2/assignment_loss‚0.30086570025378023 0loss/SRqqt1/detection_loss‚2.455374235107053 _step‚974 /loss/FRt3/assignment_loss‚1.409757681764089 1loss/FBt1/assignment_loss‚0.12867408321471813 1loss/SRqqt3/detection_loss‚1.1215157243028535 .loss/FBt2/detection_loss‚3.379006549715996 1loss/SRqqt2/assignment_loss‚1.280392823291236 .loss/FRt2/detection_loss‚2.513989941088593 1loss/SRqqt1/assignment_loss‚1.736032762174164 epoch‚9 /loss/FRt2/assignment_loss‚3.238131770962163 2loss/SRqqt3/assignment_loss‚0.8318028625227132 /loss/FRt3/detection_loss‚1.6361391195454276Î>Ž- ’°*‚(:‡ ¹‘äÔÊ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çÉI¥‚±*‚(:÷ ¹‘äÔƒ‹  disk.drbd1716.out‚0 disk.sda3.in‚ 0.1953125 cpu‚25.339985260925385% disk./.usageGB‚248.75662231445312 disk.rbd1.in‚0 disk.sda.out‚ 413.89453125 disk.sda2.in‚0 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚16.694265644231283 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 network.sent‚ 2.9257602e+07& disk.nvme0n1.out‚57664.76416015625# disk.dm-0.out‚57664.76416015625 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1187.62109375$ memory_percent‚80.41706085205078 disk.rbd1.out‚36.67333984375 disk.dm-1.out‚ 413.89453125& proc.memory.availableMB‚ 3208.46875 proc.cpu.threads‚17 network.recv‚ 3.391158e+06 disk.drbd1716.in‚0 disk.sda3.out‚ 413.89453125( disk.nvme0c0n1.out‚57664.76416015625# proc.memory.rssMB‚ 2724.73046875( proc.memory.percent‚16.6304349899292! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2080.in‚0íY ’²*‚(:‡ È‘äÔ¼»“ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éqãß³*‚(:î È‘äÔÜ Ì  network.recv‚ 3.397521e+06 disk.drbd2266.in‚0 disk.sda.out‚ 414.40625 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.out‚0 disk.sda3.out‚ 414.40625 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375& disk.nvme0n1.out‚57719.63134765625 disk.sda1.in‚0 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.dm-0.in‚ 1187.62109375* disk./.usagePercent‚16.693840415716913% disk./.usageGB‚248.75028610229492 disk.dm-1.out‚ 414.40625 disk.rbd0.in‚0 disk.sda1.out‚0# disk.dm-0.out‚57719.63134765625 cpu‚25.339513513634568 disk.sda.in‚ 0.1953125 disk.sda2.in‚0( disk.nvme0c0n1.out‚57719.63134765625 disk.rbd1.in‚0 network.sent‚ 2.9365611e+07 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚36.67333984375& proc.memory.availableMB‚ 3208.46875 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.Šp|Fšsda2.out‚0$ memory_percent‚80.41706085205078# proc.memory.rssMB‚ 2724.73046875( proc.memory.percent‚16.6304349899292 proc.cpu.threads‚17è1$—´*‚(:’ בäÔ©Ðü gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.255' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.050999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Zð—‚µ*‚(:÷ בäÔ…´Ž * disk./.usagePercent‚16.696324456821646% disk./.usageGB‚248.78730010986328 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚36.67333984375' proc.memory.availableMB‚ 3207.796875 network.recv‚ 3.401893e+06 disk.sda3.out‚ 414.76171875 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda.out‚ 414.76171875 disk.rbd0.in‚0! disk.nvme0n1.in‚ 1187.62109375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 414.76171875 disk.drbd2265.in‚0 cpu‚25.33905479383221 proc.cpu.threads‚17 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 network.sent‚ 2.9371724e+07 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚57780.06884765625$ memory_percent‚80.42116165161133( disk.nvme0c0n1.out‚57780.06884765625 disk.rbd1.in‚0# proc.memory.rssMB‚ 2724.73046875 disk.sda3.in‚ 0.1953125 disk.drbd2080.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚57780.06884765625 disk.drbd2265.out‚0 disk.sda.in‚ 0.1953125( proc.memory.percent‚16.6304349899292# disk.nvme0c0n1.in‚ 1187.62109375n6—2¶*- _wandbruntime‚9274  _runtime‚92747ñsÈ·* & lr-AdamW/pg2‚2.090666416330336e-07 trainer/global_step‚11899 % _timestamp‚1.7884141716384416e+09 _runtime‚9274.625931376 _step‚975 & lr-AdamW/pg1‚2.090666416330336e-07‹[ܙ͸*Ç _step‚975 & lr-AdamW/pg1‚2.090666416330336e-07 & lr-AdamW/pg2‚2.090666416330336e-07 trainer/global_step‚11899 % _timestamp‚1.7884141716384416e+09 _runtime‚9274.625931376Ï‹¼»,2¹*- _wandbruntime‚9274  _runtime‚9274i±ƒì'º*æO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚755139950469632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172924998 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924879188175 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701462233696768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9962206955520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157697334 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚755139974409728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9961550593024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224572480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224552442 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701462908062208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172925000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673386398682112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚9274.626719985 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397477440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚701462908062208 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172924998 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224572480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9961550593024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚691500701106688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701462233696768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56804664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691500701106688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157696886 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10449620761600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚683311949347403 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924879188175 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701462233696768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9962206955520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157697334 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚755139974409728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9961550593024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224572480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224552442 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701462908062208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172925000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673386398682112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚9274.626719985 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397477440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚701462908062208 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172924998 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224572480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9961550593024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚691500701106688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701462233696768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56804664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691500701106688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157696886 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10449620761600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚683311949347403 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9925695623139 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683367669913135 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172939016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚701520080542208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178290688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9963026396672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚691557036142592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673441302812672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157710071 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56809233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157709612 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56808776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100900838 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚977 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701519406163456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673441319852032 _runtime‚9275.335088696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224570896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚755201601207296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10450472693248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691557054145536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224570896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224590960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9962370020864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884141723469162e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172939014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765651895609856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚683366998435811 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100900836 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172939014 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚765652097840640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9926350061103 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691557054145536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚755201625147392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9962370020864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 39750÷¿u9910z 4Ñû&¾*õM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172939014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765651895609856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚683366998435811 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100900836 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172939014 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚765652097840640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9926350061103 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691557054145536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚755201625147392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9962370020864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397509910 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9925695623139 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683367669913135 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172939016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚701520080542208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178290688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9963026396672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚691557036142592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673441302812672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157710071 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56809233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157709612 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56808776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100900838 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚977 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701519406163456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673441319852032 _runtime‚9275.335088696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224570896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚755201601207296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10450472693248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691557054145536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224570896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224590960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9962370020864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884141723469162e+09ѳYé«2¿*- _wandbruntime‚9275  _runtime‚9275¡åù]À*× /loss/SRqqt1/detection_loss‚2.27826463235052 .loss/FBt1/detection_loss‚3.830394521355629 % _timestamp‚1.7884141723488615e+09 /loss/FRt1/assignment_loss‚3.350535405093226 2loss/SRqqt1/assignment_loss‚1.3218144848942757 1loss/SRqqt4/detection_loss‚1.7527107914101403 /loss/FRt1/detection_loss‚2.7988253153603653 _step‚978 1loss/FBt2/assignment_loss‚0.23719205010671762 2loss/SRqqt2/assignment_loss‚1.4044928562275514 1loss/FBt1/assignment_loss‚0.01226685184519738 %loss/total_loss‚41.58980397176806 epoch‚9 1loss/FBt3/assignment_loss‚0.13128057658734626 .loss/FRt2/detection_loss‚2.420409423912598 /loss/FRt4/detection_loss‚2.4130239844322205 _runtime‚9275.336220673 0loss/FRt2/assignment_loss‚3.0038813236283093 /loss/FRt4/assignment_loss‚2.385553154349327 0loss/SRqqt2/detection_loss‚1.747151857310975 /loss/FBt2/detection_loss‚2.3020961609753696 /loss/FBt4/detection_loss‚2.6018106937408447 1loss/SRqqt3/assignment_loss‚0.889049034233072 1loss/FBt4/assignment_loss‚0.01827215235632846 /loss/FRt3/detection_loss‚1.8081246716125559 1loss/SRqqt3/detection_loss‚0.9503827429642635 0loss/FBt3/detection_loss‚0.49624022102283166 0loss/FRt3/assignment_loss‚1.9401874554653962 2loss/SRqqt4/assignment_loss‚1.4958436105330035 trainer/global_step‚11899šÌúbÁ*Ü 1loss/SRqqt3/assignment_loss‚0.889049034233072 1loss/FBt4/assignment_loss‚0.01827215235632846 /loss/FRt3/detection_loss‚1.8081246716125559 1loss/SRqqt3/detection_loss‚0.9503827429642635 0loss/FBt3/detection_loss‚0.49624022102283166 0loss/FRt3/assignment_loss‚1.9401874554653962 2loss/SRqqt4/assignment_loss‚1.4958436105330035 trainer/global_step‚11899 /loss/SRqqt1/detection_loss‚2.27826463235052 .loss/FBt1/detection_loss‚3.830394521355629 % _timestamp‚1.7884141723488615e+09 /loss/FRt1/assignment_loss‚3.350535405093226 2loss/SRqqt1/assignment_loss‚1.3218144848942757 1loss/SRqqt4/detection_loss‚1.7527107914101403 /loss/FRt1/detection_loss‚2.7988253153603653 _step‚978 1loss/FBt2/assignment_loss‚0.23719205010671762 2loss/SRqqt2/assignment_loss‚1.4044928562275514 1loss/FBt1/assignment_loss‚0.01226685184519738 %loss/total_loss‚41.58980397176806 epoch‚9 1loss/FBt3/assignment_loss‚0.13128057658734626 .loss/FRt2/detection_loss‚2.420409423912598 /loss/FRt4/detection_loss‚2.4130239844322205 _runtime‚9275.336220673 0loss/FRt2/assignment_loss‚3.0038813236283093 /loss/FRt4/assignment_loss‚2.385553154349327 0loss/SRqqt2/detection_loss‚1.747151857310975 /loss/FBt2/detection_loss‚2.3020961609753696 /loss/FBt4/detection_loss‚2.6018106937408447Ò Q–Â*‚(:’ æ‘äÔŹž gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.472' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.294400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É_¢YƒÃ*‚(:ø æ‘äÔ¥½¤  disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125 disk.sda.out‚ 415.41796875 disk.drbd1716.out‚0 disk.rbd1.out‚36.75927734375! disk.nvme0n1.in‚ 1187.62109375# disk.dm-0.out‚57818.75244140625 disk.dm-1.out‚ 415.41796875 disk.sda3.out‚ 415.41796875( proc.memory.percent‚16.6304349899292* disk./.usagePercent‚16.697428105301757 disk.dm-0.in‚ 1187.62109375 network.recv‚ 3.406081e+06& disk.nvme0n1.out‚57818.75244140625 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚ 2.9377618e+07( disk.nvme0c0n1.out‚57818.75244140625 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0$ memory_percent‚80.41985034942627 cpu‚25.33858245389811 disk.sda2.in‚0) proc.memory.availableMB‚ 3208.01171875 disk.rbd1.in‚0 disk.sda3.in‚ 0.1953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0$ disk./.usageGB‚248.8037452697754# disk.nvme0c0n1.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125 disk.sda2.out‚0# proc.memory.rssMB‚ 2724.73046875 proc.cpu.threads‚17˜ ›ÛÄ*‚(:’ õ‘äÔÕ»” gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.319' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.263799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>íáÅ*‚(:ô õ‘äÔ€’Œ  disk.drbd2265.in‚0 disk.dm-0.in‚ 1187.62109375 disk.drbd2080.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1187.62109375! disk.nvme0n1.in‚ 1187.62109375# proc.memory.rssMB‚ 2724.73046875 network.recv‚ 3.412845e+06 disk.sda.out‚ 415.7734375 disk.drbd2266.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚57894.30908203125 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125% disk./.usageGB‚248.82448196411133 disk.sda3.out‚ 415.7734375 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.drbd2265.out‚0( proc.memory.percent‚16.6304349899292 network.sent‚ 2.9486791e+07* disk./.usagePercent‚16.698819762257884 disk.sda.in‚ 0.1953125 disk.rbd0.out‚0 disk.drbd1716.out‚0 cpu‚25.33812498158504 disk.sda3.in‚ 0.1953125 disk.rbd1.out‚36.82177734375# disk.dm-0.out‚57894.30908203125$ memory_percent‚80.41982650756836' proc.memory.availableMB‚ 3208.015625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚57894.30908203125 disk.dm-1.out‚ 415.7734375¸†2Æ*- _wandbruntime‚9311  _runtime‚9311iQ¹;ÈÇ* _step‚979 & lr-AdamW/pg1‚1.139478976367415e-07 & lr-AdamW/pg2‚1.139478976367415e-07 trainer/global_step‚11949 % _timestamp‚1.7884142088533309e+09 _runtime‚9311.840808876êƒÜŽÍÈ*Ç _runtime‚9311.840808876 _step‚979 & lr-AdamW/pg1‚1.139478976367415e-07 & lr-AdamW/pg2‚1.139478976367415e-07 trainer/global_step‚11949 % _timestamp‚1.7884142088533309e+09Ó]- ¼2É*- _wandbruntime‚9311  _runtime‚9311§Võ~ï'Ê*éO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚419 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚676131622249472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9966368965339 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704320871444992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚694318353049088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚758222511292928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158334070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57032581 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173625800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10492493668864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 399104292 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚686097991214811 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57033000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚11949 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚704321545810432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173625800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚676131605210112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768714981021696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 399124332 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101301068 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704320871444992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10003192761344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376¶âZéƒË*îO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚11949 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚704321545810432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173625800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚676131605210112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768714981021696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 399124332 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101301068 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704320871444992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10003192761344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚419 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚676131622249472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9966368965339 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704320871444992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚694318353049088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚758222511292928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158334070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57032581 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173625800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10492493668864 F8DeviceStatsMonitor.on_train²g°Ûq_batch_start/active.all.freed‚ 399104292 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚686097991214811 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57033000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156828 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11949 $ _timestamp‚1.788414209535722e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚694374706087936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768777506424832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676186509340672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158346307 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚981 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚704378718290432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225496948 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173639814 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚758284138090496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9966530962547 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399136762 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚704378718290432 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158346749 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399156828 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225517012 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚758284162030592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10493368334336 _runtime‚9312.524137571 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚686153040303219 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10003355826688 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225517012 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 399136762 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101309236 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872æÏ„Ðþ&Î*øM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676186509340672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158346307 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚981 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚704378718290432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225496948 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173639814 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚758284138090496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9966530962547 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399136762 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚704378718290432 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158346749 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399156828 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225517012 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚758284162030592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10493368334336 _runtime‚9312.524137571 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚686153040303219 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10003355826688 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225517012 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 399136762 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101309236 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156828 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11949 $ _timestamp‚1.788414209535722e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚694374706087936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768777506424832ÕÅß™2Ï*- _wandbruntime‚9312  _runtime‚9312ߘÀwdÐ*Þ 2loss/SRqqt1/assignment_loss‚1.8228822171136185 1loss/FBt1/assignment_loss‚0.11563498533279118 0loss/FRt4/assignment_loss‚2.6476715207099915 2loss/SRqqt4/assignment_loss‚1.4211770965427648 1loss/FBt3/assignment_loss‚0.05725345911568258 .loss/FBt2/detection_loss‚2.394076340728336 trainer/global_step‚11949 0loss/FRt1/assignment_loss‚3.0272172311941783 0loss/FRt3/assignment_loss‚0.9874286674536191 2loss/SRqqt3/assignment_loss‚0.9667438606359064 &loss/total_loss‚39.319207096379074 .loss/FRt1/detection_loss‚2.908660298585892 1loss/SRqqt3/detection_loss‚0.6267595171183348 1loss/SRqqt4/detection_loss‚1.6234934819770641 /loss/FBt4/detection_loss‚3.2997068762779236 % _timestamp‚1.7884142095382783e+09 0loss/FRt2/assignment_loss‚2.5763275139033794 2loss/FBt2/assignment_loss‚0.008499239778353108 2loss/FBt4/assignment_loss‚0.005007163765835035 .loss/FRt2/detection_loss‚2.261721789762322 _runtime‚9312.525639096 /loss/FRt3/detection_loss‚1.4064847173599095 1loss/SRqqt1/detection_loss‚2.6833881404664783 /loss/FBt1/detection_loss‚2.5287443134519787 epoch‚9 _step‚982 /loss/FRt4/detection_loss‚2.2127094831106797 1loss/SRqqt2/detection_loss‚2.0087994686431356 /loss/FBt3/detection_loss‚0.4728084538364783 2loss/SRqqt2/assignment_loss‚1.2560112595144246VDGiÑ*ã 2loss/SRqqt2/assignment_loss‚1.2560112595144246 2loss/SRqqt1/assignment_loss‚1.8228822171136185 1loss/FBt1/assignment_loss‚0.11563498533279118 0loss/FRt4/assignment_loss‚2.6476715207099915 2loss/SRqqt4/assignment_loss‚1.4211770965427648 1loss/FBt3/assignment_loss‚0.05725345911568258 .loss/FBt2/detection_loss‚2.394076340728336 trainer/global_step‚11949 0loss/FRt1/assignment_loss‚3.0272172311941783 0loss/FRt3/assignment_loss‚0.9874286674536191 2loss/SRqqt3/assignment_loss‚0.9667438606359064 &loss/total_loss‚39.319207096379074 .loss/FRt1/detection_loss‚2.908660298585892 1loss/SRqqt3/detection_loss‚0.6267595171183348 1loss/SRqqt4/detection_loss‚1.6234934819770641 /loss/FBt4/detection_loss‚3.2997068762779236 % _timestamp‚1.7884142095382783e+09 0loss/FRt2/assignment_loss‚2.5763275139033794 2loss/FBt2/assignment_loss‚0.008499239778353108 2loss/FBt4/assignment_loss‚0.005007163765835035 .loss/FRt2/detection_loss‚2.261721789762322 _runtime‚9312.525639096 /loss/FRt3/detection_loss‚1.4064847173599095 1loss/SRqqt1/detection_loss‚2.6833881404664783 /loss/FBt1/detection_loss‚2.5287443134519787 epoch‚9 _step‚982 /loss/FRt4/detection_loss‚2.2127094831106797 1loss/SRqqt2/detection_loss‚2.0087994686431356 /loss/FBt3/detection_loss‚0.4728084538364783ÖÕé8’Ò*‚(:‡ „’äÔ¼Ôà gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚177.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½<…Ó*‚(:ú „’äÔÃ–Ï & disk.nvme0n1.out‚57952.18017578125 network.recv‚ 3.416522e+06 disk.drbd2080.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.rbd1.out‚36.82177734375! disk.nvme0n1.in‚ 1187.62109375# disk.dm-0.out‚57952.18017578125 disk.drbd2265.in‚0 disk.sda.in‚ 0.1953125# proc.memory.rssMB‚ 2724.73046875 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1953125 disk.drbd1716.out‚0 disk.dm-1.out‚ 416.26953125$ memory_percent‚80.41918277740479 disk.sda2.out‚0( disk.nvme0c0n1.out‚57952.18017578125 disk.drbd1716.in‚0) proc.memory.availableMB‚ 3208.12109375( proc.memory.percent‚16.6304349899292% disk./.usageGB‚248.84543228149414 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 network.sent‚ 2.9491987e+07* disk./.usagePercent‚16.700225755635934 disk.sda3.out‚ 416.26953125 disk.dm-0.in‚ 1187.62109375 cpu‚25.337653648607414 disk.drbd2080.in‚0 disk.sda.out‚ 416.26953125 proc.cpu.threads‚17õ¿é­œÔ*‚(:‘ “’äÔÛ„ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.91' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.382000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áTñ~Õ*‚(:ó “’äԭ˜ ! disk.nvme0n1.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda.out‚ 416.7265625 cpu‚25.337200424223273 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375' disk.nvme0c0n1.out‚58012.6943359375% disk.nvme0n1.out‚58012.6943359375 disk.dm-0.in‚ 1187.62109375# proc.memory.rssMB‚ 2724.73046875 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚58012.6943359375) disk./.usagePercent‚16.70162432479549 disk.sda3.out‚ 416.7265625 disk.dm-1.out‚ 416.7265625$ memory_percent‚80.42004108428955) proc.memory.availableMB‚ 3207.98046875 disk.dm-1.in‚ 0.1953125% disk./.usageGB‚248.86627197265625 disk.drbd1716.in‚0 disk.drbd2265.in‚0( proc.memory.percent‚16.6304349899292 network.sent‚ 2.9600777e+07 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚36.85400390625 disk.drbd2265.out‚0 network.recv‚ 3.423397e+06 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0ÒŸßÖ*‚(:’ ¢’äÔý©é gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.737' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.547399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÏ_r×*‚(:ç ¢’äÔ•³Ú " disk.nvme0c0n1.out‚ 58071.53125 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 3207.9765625 disk.dm-1.out‚ 417.30859375 disk.rbd0.in‚0 disk.drbd1716.out‚0% disk./.usageGB‚248.88005828857422 disk.sda.in‚ 0.1953125 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.nvme0n1.out‚ 58071.53125$ memory_percent‚80.42006492614746 disk.dm-0.out‚ 58071.53125 disk.sda.out‚ 417.30859375! disk.nvme0n1.in‚ 1187.62109375( proc.memory.percent‚16.6304349899292 cpu‚25.336737796207554 disk.sda3.out‚ 417.30859375 network.sent‚ 2.9606684e+07 network.recv‚ 3.427619e+06 disk.drbd2265.out‚0 disk.rbd1.out‚36.85400390625# proc.memory.rssMB‚ 2724.73046875 disk.dm-0.in‚ 1187.62109375 disk.dm-1.in‚ 0.1953125# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd1716.in‚0* disk./.usagePercent‚16.702549536024232 disk.drbd2080.in‚0 disk.drbd2080.out‚0øØÚ2Ø*- _wandbruntime‚9348  _runtime‚9348Ï›Ò|ÇÙ*Á & lr-AdamW/pg1‚4.746635392614367e-08 & lr-AdamW/pg2‚4.746635392614367e-08 trainer/global_step‚11999 $ _timestamp‚1.788414245685584e+09 _runtime‚9348.672995166 _step‚983V‘óÌÚ*Æ _step‚983 & lr-AdamW/pg1‚4.746635392614367e-08 & lr-AdamW/pg2‚4.746635392614367e-08 trainer/global_step‚11999 $ _timestamp‚1.788414245685584e+09 _runtime‚9348.672995166×wA2Û*- _wandbruntime‚9348  _runtime‚9348M9ÐS²Ü*ìO trainer/global_step‚11999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10044178567168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚771840718209536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.a ¼åc@&ll.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226404546 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚707179509193216 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚707180183558656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10535500118016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707179509193216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400751184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚697135986988544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697135986988544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101709468 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10043522204672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697136004991488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10043522204672 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158970242 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174326598 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707180183558656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688884033082219 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚678876828777472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761305048176128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚678876811738112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400731144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884142456862614e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57260774 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚771840524354048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10044178567168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174326600 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚761305024236032 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10006549866991÷ TÕ÷'Ý*ñO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400751184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚697135986988544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697135986988544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101709468 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10043522204672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697136004991488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10043522204672 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158970242 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174326598 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707180183558656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688884033082219 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚678876828777472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761305048176128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚678876811738112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400731144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884142456862614e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57260774 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚771840524354048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10044178567168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174326600 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚761305024236032 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10006549866991 trainer/global_step‚11999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10044178567168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚771840718209536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226404546 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚707179509193216 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚707180183558656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10535500118016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707179509193216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Ø;Š]2Þ*- _wandbruntime‚9349  _runtime‚9349~°+yü&ß*öM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚697192340027392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101717636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚707236681659904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚761366674973696 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174340616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158982942 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174340616 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚761366698913792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101717638 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10044341632512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884142463811724e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763614 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10044998008320 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚697192358030336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174340614 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚771903050686976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚678931715868672 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226423000 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400763614 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771903229849088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688939753647951 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678931732908032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚697192340027392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57265737 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚9349.369025922 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174340614 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10044998008320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10007366301955 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚11999 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10536530935296y’Ôìx à*ûM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101717638 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10044341632512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884142463811724e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763614 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10044998008320 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚697192358030336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174340614 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚771903050686976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚678931715868672 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226423000 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400763614 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771903229849088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688939753647951 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678931732908032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚697192340027392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57265737 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚9349.369025922 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174340614 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10044998008320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10007366301955 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚11999 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10536530935296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚697192340027392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101717636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚707236681659904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚761366674973696 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174340616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158982942 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174340616 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚761366698913792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752ÙÓ¶”2á*- _wandbruntime‚9349  _runtime‚9349ëCaâ*Û _step‚986 0loss/FRt4/assignment_loss‚2.9062429640028213 1loss/FBt2/assignment_loss‚0.11328741477336735 /loss/FRt3/detection_loss‚1.7826714547617095 /loss/FBt4/detection_loss‚1.8335172235965729 .loss/FRt4/detection_loss‚2.275481175714069 1loss/SRqqt1/detection_loss‚2.3987989097833635 _runtime‚9349.370252627 0loss/FRt2/assignment_loss‚2.5759710552899735 0loss/FRt3/assignment_loss‚1.2978336823229888 2loss/SRqqt3/assignment_loss‚0.7816077908195678 1loss/SRqqt2/detection_loss‚1.7686195662792992 /loss/FBt3/detection_loss‚0.6090738592209601 0loss/SRqqt3/detection_loss‚1.110937015504622 .loss/FBt1/detection_loss‚2.615821987390518 /loss/FBt2/detection_loss‚2.2407087087631226 1loss/FBt4/assignment_loss‚0.43771366622422647 2loss/SRqqt2/assignment_loss‚1.3299414026824867 1loss/FBt1/assignment_loss‚0.23344713004189543 % _timestamp‚1.7884142463828704e+09 2loss/SRqqt1/assignment_loss‚1.2241142178419977 epoch‚9 2loss/SRqqt4/assignment_loss‚1.2173381243612513 1loss/SRqqt4/detection_loss‚1.6850858770862773 trainer/global_step‚11999 0loss/FRt1/assignment_loss‚2.3364050447940827 1loss/FBt3/assignment_loss‚0.06749007731044984 /loss/FRt1/detection_loss‚2.9428633292516073 /loss/FRt2/detection_loss‚2.0856495648334015 %loss/total_loss‚37.87062124265063±ÜKfã*à 0loss/SRqqt3/detection_loss‚1.110937015504622 .loss/FBt1/detection_loss‚2.615821987390518 /loss/FBt2/detection_loss‚2.2407087087631226 1loss/FBt4/assignment_loss‚0.43771366622422647 2loss/SRqqt2/assignment_loss‚1.3299414026824867 1loss/FBt1/assignment_loss‚0.23344713004189543 % _timestamp‚1.7884142463828704e+09 2loss/SRqqt1/assignment_loss‚1.2241142178419977 epoch‚9 2loss/SRqqt4/assignment_loss‚1.2173381243612513 1loss/SRqqt4/detection_loss‚1.6850858770862773 trainer/global_step‚11999 0loss/FRt1/assignment_loss‚2.3364050447940827 1loss/FBt3/assignment_loss‚0.06749007731044984 /loss/FRt1/detection_loss‚2.9428633292516073 /loss/FRt2/detection_loss‚2.0856495648334015 %loss/total_loss‚37.87062124265063 _step‚986 0loss/FRt4/assignment_loss‚2.9062429640028213 1loss/FBt2/assignment_loss‚0.11328741477336735 /loss/FRt3/detection_loss‚1.7826714547617095 /loss/FBt4/detection_loss‚1.8335172235965729 .loss/FRt4/detection_loss‚2.275481175714069 1loss/SRqqt1/detection_loss‚2.3987989097833635 _runtime‚9349.370252627 0loss/FRt2/assignment_loss‚2.5759710552899735 0loss/FRt3/assignment_loss‚1.2978336823229888 2loss/SRqqt3/assignment_loss‚0.7816077908195678 1loss/SRqqt2/detection_loss‚1.7686195662792992 /loss/FBt3/detection_loss‚0.6090738592209601ÚÅšËä*‚(:’ ±’äÔÏò– gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.339' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.867799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k2{å*‚(:ð ±’äÔÜ ³  disk.drbd2080.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1187.62109375& disk.nvme0n1.out‚58150.06982421875# disk.dm-0.out‚58150.06982421875 disk.sda3.out‚ 417.859375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2724.73046875 network.recv‚ 3.43208e+06 disk.drbd2080.out‚0# memory_percent‚80.4203748703003) proc.memory.availableMB‚ 3207.92578125( proc.memory.percent‚16.6304349899292 disk.sda.out‚ 417.859375 cpu‚25.336281209771656) disk./.usagePercent‚16.70385875855434 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 417.859375( disk.nvme0c0n1.out‚58150.06982421875 disk.rbd1.out‚ 36.9169921875 disk.dm-1.in‚ 0.1953125 disk.sda.in‚ 0.1953125 disk.sda3.in‚ 0.1953125 disk.sda1.in‚0% disk./.usageGB‚248.89956665039062 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 network.sent‚ 2.9613003e+07 disk.rbd1.in‚0 disk.dm-0.in‚ 1187.62109375*…³ã’æ*‚(:‡ À’äÔÄ…ˆ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.483' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8966 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ƒc{„ç*‚(:ù À’äÔꤘ  network.recv‚ 3.438721e+06 disk.drbd2265.in‚0 disk.sda3.out‚ 418.36328125 disk.sda1.out‚0) proc.memory.availableMB‚ 3207.91796875 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1187.62109375& disk.nvme0n1.out‚58210.72998046875 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 36.9169921875$ memory_percent‚80.42042255401611 cpu‚25.335819980843134 disk.dm-1.in‚ 0.1953125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚58210.72998046875 disk.drbd1716.in‚0 network.sent‚ 2.9721405e+07 disk.dm-0.in‚ 1187.62109375# disk.dm-0.out‚58210.72998046875 disk.sda.in‚ 0.1953125 disk.sda.out‚ 418.36328125 disk.sda3.in‚ 0.1953125 disk.sda2.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚16.704834403267327% disk./.usageGB‚248.91410446166992 disk.dm-1.out‚ 418.36328125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2724.73046875( proc.memory.percent‚16.6304349899292¢7û2è*- _wandbruntime‚9385  _runtime‚9385M‘P¸Èé* & lr-AdamW/pg1‚9.635865155328637e-09 & lr-AdamW/pg2‚9.635865155328637e-09 trainer/global_step‚12049 % _timestamp‚1.7884142827082005e+09 _runtime‚9385.696091836 _step‚987[býCÍê*Ç & lr-AdamW/pg1‚9.635865155328637e-09 & lr-AdamW/pg2‚9.635865155328637e-09 trainer/global_step‚12049 % _timestamp‚1.7884142827082005e+09 _runtime‚9385.696091836 _step‚987Û`Õ®±2ë*- _wandbruntime‚9385  _runtime‚9385YÄøò'ì*ìO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884142827092977e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10047385206399 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710038146941440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚764387561119232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227330598 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _runtime‚9385.697005851 _step‚988 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699953656933888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚691669403472511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57487920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚699953638930944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10084508010496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175027398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚691670074949627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102117868 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681622035305472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402357996 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175027400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10084508010496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402378036 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159605788 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10085164372992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10085164372992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10048039644155 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚699953656933888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227350636 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57488366 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175027400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710038146941440 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681622018266112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175027398 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710038821306880 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102117870 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚764387585059328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10578590680064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227350636 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159606236 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ÈîVþ"í*ñO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚699953656933888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227350636 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57488366 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175027400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710038146941440 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681622018266112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175027398 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710038821306880 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102117870 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚764387585059328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10578590680064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227350636 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159606236 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884142827092977e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10047385206399 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710038146941440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚764387561119232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227330598 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _runtime‚9385.697005851 _step‚988 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699953656933888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚691669403472511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57487920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚699953638930944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10084508010496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175027398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚691670074949627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102117868 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681622035305472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402357996 QCDeviceStatsMoni&‰V¿ôtor.on_train_batch_start/active.large_pool.allocated‚ 175027400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10084508010496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402378036 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159605788 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10085164372992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10085164372992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10048039644155ÜŒP¥X2î*- _wandbruntime‚9386  _runtime‚9386PJü&ï*öM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102126036 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175041416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700010009972736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085983814144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚700009991969792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10085327438336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚691725124038035 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102126038 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710095993786880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227349052 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710095993786880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159618951 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775028493812736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710095319408128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚691725795515359 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159618500 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402390466 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57492464 _step‚989 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700009991969792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681676939436032 trainer/global_step‚12049 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227369116 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10579281955840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚681676922396672 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928À9¡'ð*ûM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚9386.394671935 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10048856079327 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402390466 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57492913 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10579451857920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚764449211856896 % _timestamp‚1.7884142834065728e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227369116 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175041414 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10085983814144 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102126036 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175041416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700010009972736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085983814144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚700009991969792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10085327438336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚691725124038035 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102126038 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710095993786880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227349052 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710095993786880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159618951 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775028493812736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710095319408128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚691725795515359 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159618500 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402390466 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57492464 _step‚989 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700009991969792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681676939436032 trainer/global_step‚12049 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227369116 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10579281955840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚681676922396672 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 160663624 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 176131550 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚10136182834176 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 404581963 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 228466392 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚57892170 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚714204967769088 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115343872 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10631876956672 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚685602498873344 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 176131550 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚57892014 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 160663466 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚695701345695171 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚704068784934912 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 176131548 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _runtime‚9464.015909216 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 404581963 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚714205478602240 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚704068802937856 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚768771152305152 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 102771452 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 160663624 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 176131550 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚10136182834176 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 404581963 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 228466392 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚57892170 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚714204967769088 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115343872 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10631876956672 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚685602498873344 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 176131550 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚57892014 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 160663466 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚695701345695171 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚704068784934912 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 176131548 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _runtime‚9464.015909216 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 404581963 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚714205478602240 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚704068802937856 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚768771152305152 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872ßbËá22+- _wandbruntime‚9464  _runtime‚9464Ÿob)‚+ÜR JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚714242162924544 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404600071 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚704105632966656 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 176142354 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚779441270121984 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788414361970656e+09 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚685637775593472 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10099227470595 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10137040791040 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚10136547960832 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10099718384987 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚9464.958461869 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 228457717 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 404616050 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚779441409405952 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10632357427200 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 176142354 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 228473694 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚704105614963712 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 102777956 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚57896524 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 228457717 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 228473694 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚158 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚704105632966656 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 160674480 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 176142356 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚768809028038656 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚695737511017819 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚695737003064067 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚10137040791040 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚704105614963712 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚714242673757696 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 176142356 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚768809051978752 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚685637792632832 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10632242083328 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚714242162924544 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944â+wÔsƒ+áR OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10099227470595 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10137040791040 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚10136547960832 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.curreÔà—Óô%nt‚23940096 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10099718384987 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚9464.958461869 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 228457717 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 404616050 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚779441409405952 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10632357427200 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 176142354 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 228473694 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚704105614963712 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 102777956 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚57896524 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 228457717 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 228473694 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚158 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚704105632966656 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 160674480 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 176142356 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚768809028038656 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚695737511017819 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚695737003064067 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚10137040791040 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚704105614963712 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚714242673757696 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 176142356 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚768809051978752 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚685637792632832 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10632242083328 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚714242162924544 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚714242162924544 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404600071 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚704105632966656 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 176142354 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚779441270121984 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788414361970656e+09 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚685637775593472 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872à¼ËI‘„+‚(:„ š“äÔÑÙ– gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.816 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖáË|…+‚(:ñ š“äÔµ…•  disk.drbd2080.out‚0 disk.sda2.out‚0 network.sent‚ 2.9858744e+07 disk.rbd1.in‚0 disk.dm-0.in‚ 1187.62109375 disk.sda.in‚ 0.1953125 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.1953125 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 3130.29296875 disk.rbd1.out‚37.16064453125! disk.nvme0n1.in‚ 1187.62109375! disk.dm-0.out‚58606.013671875 disk.rbd0.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚16.708092099140536# disk./.usageGB‚248.962646484375 disk.drbd1716.in‚0 disk.dm-1.out‚ 421.4296875 disk.sda.out‚ 421.4296875 disk.drbd2266.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2801.88671875 disk.sda3.out‚ 421.4296875& disk.nvme0c0n1.out‚58606.013671875* proc.memory.percent‚17.101359367370605 cpu‚25.358346520838104$ disk.nvme0n1.out‚58606.013671875 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1187.62109375$ memory_percent‚80.89420795440674 network.recv‚ 3.465622e+06 disk.drbd2080.in‚06¤ê˜2†+- _wandbruntime‚9477  _runtime‚9477 È;¬S‡+»¦ ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016817331314086914 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17711123824119568 particle/accuracy_3_of_5‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 epoch‚9 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49513494968414307 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26724550127983093 jet/accuracy_4_of_7‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4574712812900543 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16339285671710968 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_2‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39291930198669434 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5129204988479614 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 *jet/accuracy_2_of_6‚0.4166666865348816 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9923826456069946 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3593812584877014 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4556694030761719 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24887385964393616 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22122728824615479 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +jet/accuracy_4_of_6‚0.03846153989434242 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283686965703964 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4567309617996216 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912302494049072 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45615530014038086 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 particle/accuracy_3_of_7‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12260536849498749 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1612277626991272 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 _step‚993 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.38036414980888367 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2222222089767456 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16567903757095337 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.49814820289611816 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.3248843848705292 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15269707143306732 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,jet/accuracy_4_of_5‚0.018458781763911247 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23925912380218506 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6508833765983582 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47414422035217285 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4440898895263672 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.015350878238677979 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.24368803203105927 jet/accuracy_1_of_7‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09880952537059784 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_4‚1 7validation_average_jet_accuracy‚0.43081507086753845 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5507690906524658 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35868051648139954 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 jet/accuracy_6_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 *loss/val_total_loss‚41.166524434777585 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5461406111717224 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.003250189358368516 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5787314772605896 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5802529454231262 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4155508875846863 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4987872838973999 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25076085329055786 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.513687252998352 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.47443687915802 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06240920349955559 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.424112468957901 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21256038546562195 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18028177320957184 particle/accuracy_1_of_3‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9907804727554321 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.040028978139162064 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03137630224227905 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2776424288749695 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14742262661457062 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2778880000114441 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49408286809921265 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.007084419950842857 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5182757377624512 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3613303601741791 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18190257251262665 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1456274539232254 particle/accuracy_2_of_4‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2638888955116272 particle/accuracy_2_of_6‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03694581240415573 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16251572966575623 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5091186761856079 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8554959893226624 particle/accuracy_2_of_5‚1 (particle/accuracy‚0.8536189198493958 particle/accuracy_4_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875088334083557 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41861605644226074 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21241216361522675 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.41632989048957825 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02421383745968342 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4940257966518402 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0520833320915699 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4111607074737549 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.029647868126630783 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20444968342781067 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.09615384787321091 jet/accuracy_3_of_7‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.925409734249115 particle/accuracy_5_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4374999701976776 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6043498516082764 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1577777862548828 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_1_of_6‚1 particle/accuracy_2_of_7‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14712642133235931 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18199798464775085 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27424800395965576 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4516281485557556 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15360477566719055 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5697501301765442 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012503346428275108 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41786783933639526 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.28072527050971985 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.563218355178833 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6129606366157532 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16181661188602448 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 trainer/global_step‚12089 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14035087823867798 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16717372834682465 +jet/accuracy_3_of_4‚0.07353047281503677 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4705239534378052 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4164724051952362 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33761265873908997 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18190257251262665 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979721903800964 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11314436048269272 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23765431344509125 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 particle/accuracy_6_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10366590321063995 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14269743859767914 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9686892032623291 +jet/accuracy_3_of_5‚0.11262160539627075 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02824500948190689 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.013565892353653908 +particle/specificity‚0.9514623284339905 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026519067585468292 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.74¼]Q –?30844306945801 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933608770370483 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.695293128490448 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15302982926368713 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27339452505111694 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2489892691373825 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319833755493 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2467964142560959 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5149928331375122 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4617113471031189 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5334270000457764 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5125706195831299 5 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.98167884349823 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19056691229343414 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1846434623003006 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9913367629051208 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4767269492149353 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5561609864234924 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34876543283462524 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011263428255915642 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4931052029132843 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2084140181541443 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.540197491645813 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17927829921245575 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9926718473434448 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 *jet/accuracy_2_of_4‚0.3585379123687744 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14907053112983704 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4892991781234741 'particle/f_score‚0.4091939330101013 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.38333335518836975 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3454318344593048 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4697090983390808 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1150793582201004 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1188964694738388 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9952948093414307 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 jet/accuracy_5_of_6‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16603094339370728 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19482949376106262 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41876012086868286 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5261747241020203 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46723616123199463 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5755032896995544 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15703877806663513 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 -jet/accuracy_5_of_5‚0.0026881720405071974 jet/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 _runtime‚9477.746629673 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019312169402837753 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915906190872192 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5017520189285278 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4593832790851593 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21724137663841248 *jet/accuracy_2_of_5‚0.4379480481147766 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39128535985946655 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5698413252830505 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9820917248725891 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5678571462631226 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08362264931201935 particle/accuracy_3_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16241177916526794 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42203351855278015 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.289814829826355 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988924264907837 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06923288106918335 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2013889104127884 -"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5625 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9915334582328796 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8205224275588989 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3946760892868042 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5047373175621033 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4242337942123413 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 % _timestamp‚1.7884143747571845e+09 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5537037253379822 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22042079269886017 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12325304001569748 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02313969098031521 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 particle/accuracy_1_of_7‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9928174614906311 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3992248475551605 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39333000779151917 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20776206254959106 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19765740633010864 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9900063872337341 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.607291579246521 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22412312030792236 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005095345433801413 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.45504385232925415 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4760756492614746 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34464287757873535 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.496921181678772 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28636136651039124 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5303431153297424 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02893758751451969 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4870230257511139 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47443687915802 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.040364257991313934 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39537033438682556 ("Purity/*FRt2SRqqt2FBt/event_purity‚0˜¨ÎÒ\@ˆ+À¦ 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4593832790851593 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21724137663841248 *jet/accuracy_2_of_5‚0.4379480481147766 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39128535985946655 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5698413252830505 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9820917248725891 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5678571462631226 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08362264931201935 particle/accuracy_3_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16241177916526794 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42203351855278015 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.289814829826355 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988924264907837 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06923288106918335 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2013889104127884 -"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5625 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9915334582328796 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8205224275588989 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3946760892868042 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5047373175621033 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4242337942123413 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 % _timestamp‚1.7884143747571845e+09 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5537037253379822 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22042079269886017 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12325304001569748 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02313969098031521 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 particle/accuracy_1_of_7‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9928174614906311 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3992248475551605 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39333000779151917 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20776206254959106 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19765740633010864 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9900063872337341 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.607291579246521 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22412312030792236 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005095345433801413 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.45504385232925415 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4760756492614746 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34464287757873535 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.496921181678772 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28636136651039124 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5303431153297424 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02893758751451969 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4870230257511139 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47443687915802 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.040364257991313934 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39537033438682556 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016817331314086914 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17711123824119568 particle/accuracy_3_of_5‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 epoch‚9 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49513494968414307 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26724550127983093 jet/accuracy_4_of_7‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4574712812900543 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16339285671710968 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_2‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39291930198669434 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5129204988479614 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 *jet/accuracy_2_of_6‚0.4166666865348816 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9923826456069946 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3593812584877014 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4556694030761719 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24887385964393616 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22122728824615479 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +jet/accuracy_4_of_6‚0.03846153989434242 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283686965703964 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4567309617996216 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912302494049072 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45615530014038086 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 particle/accuracy_3_of_7‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12260536849498749 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1612277626991272 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 _step‚993 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.38036414980888367 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2222222089767456 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16567903757095337 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.49814820289611816 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.3248843848705292 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15269707143306732 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,jet/accuracy_4_of_5‚0.018458781763911247 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23925912380218506 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6508833765983582 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47414422035217285 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4440898895263672 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.015350878238677979 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.24368803203105927 jet/accuracy_1_of_7‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09880952537059784 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_4‚1 7validation_average_jet_accuracy‚0.43081507086753845 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5507690906524658 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35868051648139954 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 jet/accuracy_6_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 *loss/val_total_loss‚41.166524434777585 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5461406111717224 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.003250189358368516 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5787314772605896 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5802529454231262 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4155508875846863 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4987872838973999 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25076085329055786 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.513687252998352 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.47443687915802 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06240920349955559 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.424112468957901 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21256038546562195 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18028177320957184 particle/accuracy_1_of_3‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9907804727554321 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.040028978139162064 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.64583331†ôëR34651184 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03137630224227905 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2776424288749695 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14742262661457062 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2778880000114441 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49408286809921265 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.007084419950842857 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5182757377624512 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3613303601741791 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18190257251262665 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1456274539232254 particle/accuracy_2_of_4‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2638888955116272 particle/accuracy_2_of_6‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03694581240415573 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16251572966575623 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5091186761856079 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8554959893226624 particle/accuracy_2_of_5‚1 (particle/accuracy‚0.8536189198493958 particle/accuracy_4_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875088334083557 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41861605644226074 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21241216361522675 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.41632989048957825 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02421383745968342 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4940257966518402 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0520833320915699 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4111607074737549 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.029647868126630783 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20444968342781067 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.09615384787321091 jet/accuracy_3_of_7‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.925409734249115 particle/accuracy_5_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4374999701976776 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6043498516082764 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1577777862548828 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_1_of_6‚1 particle/accuracy_2_of_7‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14712642133235931 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18199798464775085 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27424800395965576 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4516281485557556 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15360477566719055 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5697501301765442 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012503346428275108 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41786783933639526 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.28072527050971985 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.563218355178833 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6129606366157532 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16181661188602448 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 trainer/global_step‚12089 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14035087823867798 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16717372834682465 +jet/accuracy_3_of_4‚0.07353047281503677 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4705239534378052 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4164724051952362 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33761265873908997 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18190257251262665 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979721903800964 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11314436048269272 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23765431344509125 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 particle/accuracy_6_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10366590321063995 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14269743859767914 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9686892032623291 +jet/accuracy_3_of_5‚0.11262160539627075 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02824500948190689 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.013565892353653908 +particle/specificity‚0.9514623284339905 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026519067585468292 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7430844306945801 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933608770370483 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.695293128490448 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15302982926368713 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27339452505111694 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2489892691373825 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319833755493 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2467964142560959 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5149928331375122 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4617113471031189 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5334270000457764 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5125706195831299 5 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.98167884349823 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19056691229343414 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1846434623003006 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9913367629051208 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4767269492149353 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5561609864234924 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34876543283462524 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011263428255915642 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4931052029132843 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2084140181541443 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.540197491645813 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17927829921245575 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9926718473434448 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 *jet/accuracy_2_of_4‚0.3585379123687744 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14907053112983704 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4892991781234741 'particle/f_score‚0.4091939330101013 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.38333335518836975 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3454318344593048 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4697090983390808 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1150793582201004 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1188964694738388 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9952948093414307 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 jet/accuracy_5_of_6‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16603094339370728 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19482949376106262 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41876012086868286 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5261747241020203 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46723616123199463 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5755032896995544 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15703877806663513 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 -jet/accuracy_5_of_5‚0.0026881720405071974 jet/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 _runtime‚9477.746629673 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019312169402837753 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915906190872192 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5017520189285278 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408á lｎ‰+‚(:ƒ ©“äÔì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚32.0 gpu.0.powerWatts‚99.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.918 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…4ëï~Š+‚(:ó ©“äÔÁ˜¢  network.sent‚ 2.9967668e+07) disk./.usagePercent‚16.71008946992268# disk.dm-0.out‚58675.18310546875$ memory_percent‚80.89542388916016& proc.memory.availableMB‚ 3130.09375 disk.dm-1.in‚ 0.1953125 network.recv‚ 3.473057e+06 disk.dm-1.out‚ 421.91796875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.1953125 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0! disk.nvme0n1.in‚ 1187.62109375( disk.nvme0c0n1.out‚58675.18310546875 disk.rbd1.out‚37.16064453125$ disk./.usageGB‚248.9924087524414 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.1953125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.out‚ 421.91796875 disk.sda3.out‚ 421.91796875 disk.rbd0.out‚0 disk.dm-0.in‚ 1187.62109375 cpu‚25.36454551711099 disk.drbd2266.in‚0& disk.nvme0n1.out‚58675.18310546875) proc.memory.percent‚17.10195541381836 disk.rbd1.in‚0! proc.memory.rssMB‚ 2801.984375# disk.nvme0c0n1.in‚ 1187.62109375ŸÈÜSR‹+ s8hasm0rj@ ª“äÔˆÎé£0`Trainer.fit` stopped: `max_epochs=10` reached. <ÝŸŒ+ s8hasm0rjŒ «“äÔð·¬zEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:14:52 • 0:00:00 1.37it/s v_num: sm0rDÁ³’%+ s8hasm0rj «“äÔпˬ ¹F Ä2Ž+- _wandbruntime‚9484  _runtime‚9484ó÷³+‚(’ŒJ